[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:26:32 dist-test-slave-bxm0 krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:26:32 dist-test-slave-bxm0 krb5kdc[395](info): set up 2 sockets
Sep 27 21:26:32 dist-test-slave-bxm0 krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 21:26:33.042250 388 mini_kdc.cc:121] Time spent starting KDC: real 2.196s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260927 21:26:33.211151 388 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.168s user 0.000s sys 0.003s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260927 21:26:33.538256 388 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.327s user 0.000s sys 0.004s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260927 21:26:33.836176 388 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.298s user 0.002s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/test-user.keytab.
W20260927 21:26:33.960510 388 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.124s user 0.000s sys 0.004s
Sep 27 21:26:33 dist-test-slave-bxm0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:26:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:26:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:26:34.043236 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44593
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:26:34.385814 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:26:34.386142 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:26:34.386303 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:26:34.393878 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:26:34.393999 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:26:34.394084 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:26:34.394166 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:26:34.394275 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:26:34.404476 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:26:34.405961 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:26:34.407424 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:26:34.420883 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:34.420858 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:34.420949 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:26:34.421811 408 server_base.cc:1034] running on GCE node
I20260927 21:26:34.422719 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:26:34.424234 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:26:34.425454 408 hybrid_clock.cc:648] HybridClock initialized: now 1790544394425377 us; error 66 us; skew 500 ppm
I20260927 21:26:34.425872 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:26:34 dist-test-slave-bxm0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:26:34.429350 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:26:34.431159 408 webserver.cc:460] Webserver started at http://127.0.97.62:44335/ using document root <none> and password file <none>
I20260927 21:26:34.431908 408 fs_manager.cc:359] Metadata directory not provided
I20260927 21:26:34.432044 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:26:34.432365 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:26:34.435505 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "b88d86270ab84a839fa55a111280df46"
format_stamp: "Formatted at 2026-09-27 21:26:34 on dist-test-slave-bxm0"
server_key: "15bfa8a5b6f0ed3c6aa6035b365388d4"
server_key_iv: "d11f6eb7fd5e6455d3677451869f32bd"
server_key_version: "encryptionkey@0"
I20260927 21:26:34.436311 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "b88d86270ab84a839fa55a111280df46"
format_stamp: "Formatted at 2026-09-27 21:26:34 on dist-test-slave-bxm0"
server_key: "15bfa8a5b6f0ed3c6aa6035b365388d4"
server_key_iv: "d11f6eb7fd5e6455d3677451869f32bd"
server_key_version: "encryptionkey@0"
I20260927 21:26:34.441989 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260927 21:26:34.446125 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:34.447623 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:26:34.447839 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "b88d86270ab84a839fa55a111280df46"
format_stamp: "Formatted at 2026-09-27 21:26:34 on dist-test-slave-bxm0"
server_key: "15bfa8a5b6f0ed3c6aa6035b365388d4"
server_key_iv: "d11f6eb7fd5e6455d3677451869f32bd"
server_key_version: "encryptionkey@0"
I20260927 21:26:34.448058 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:26:34.463029 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:26:34.467679 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:26:34.467998 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:26:34.487807 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44593
I20260927 21:26:34.487849 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44593 every 8 connection(s)
I20260927 21:26:34.489391 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:26:34.491290 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 408
I20260927 21:26:34.491598 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:26:34.493475 388 external_mini_cluster.cc:1428] Setting key 3f95828f9cdac716408c29711c79a2fe
I20260927 21:26:34.493688 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:34.504855 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b88d86270ab84a839fa55a111280df46: Bootstrap starting.
I20260927 21:26:34.508615 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b88d86270ab84a839fa55a111280df46: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:34.510449 475 log.cc:824] T 00000000000000000000000000000000 P b88d86270ab84a839fa55a111280df46: Log is configured to *not* fsync() on all Append() calls
I20260927 21:26:34.513525 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b88d86270ab84a839fa55a111280df46: No bootstrap required, opened a new log
I20260927 21:26:34.522253 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P b88d86270ab84a839fa55a111280df46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88d86270ab84a839fa55a111280df46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44593 } }
I20260927 21:26:34.522635 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P b88d86270ab84a839fa55a111280df46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:34.522984 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P b88d86270ab84a839fa55a111280df46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b88d86270ab84a839fa55a111280df46, State: Initialized, Role: FOLLOWER
I20260927 21:26:34.523767 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P b88d86270ab84a839fa55a111280df46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88d86270ab84a839fa55a111280df46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44593 } }
Sep 27 21:26:34 dist-test-slave-bxm0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:26:34.524066 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P b88d86270ab84a839fa55a111280df46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:26:34.524250 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P b88d86270ab84a839fa55a111280df46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:26:34.524430 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b88d86270ab84a839fa55a111280df46 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:34.527755 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P b88d86270ab84a839fa55a111280df46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88d86270ab84a839fa55a111280df46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44593 } }
I20260927 21:26:34.528448 475 leader_election.cc:302] T 00000000000000000000000000000000 P b88d86270ab84a839fa55a111280df46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b88d86270ab84a839fa55a111280df46; no voters:
I20260927 21:26:34.529145 475 leader_election.cc:288] T 00000000000000000000000000000000 P b88d86270ab84a839fa55a111280df46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:26:34.529279 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b88d86270ab84a839fa55a111280df46 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:26:34.530750 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P b88d86270ab84a839fa55a111280df46 [term 1 LEADER]: Becoming Leader. State: Replica: b88d86270ab84a839fa55a111280df46, State: Running, Role: LEADER
I20260927 21:26:34.531427 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P b88d86270ab84a839fa55a111280df46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88d86270ab84a839fa55a111280df46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44593 } }
I20260927 21:26:34.531576 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P b88d86270ab84a839fa55a111280df46 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:26:34.536242 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:34.497454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58194 (local address 127.0.97.62:44593)
0927 21:26:34.498471 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:34.498489 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:34.499814 (+ 1325us) server_negotiation.cc:407] Connection header received
0927 21:26:34.509145 (+ 9331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:34.509173 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:34.510126 (+ 953us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:34.510621 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:34.513922 (+ 3301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:34.516004 (+ 2082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:34.517786 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:34.518882 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:34.519089 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:34.525520 (+ 6431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:34.525537 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:34.525559 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:34.525648 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:34.529910 (+ 4262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:34.531069 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:34.531077 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:34.531085 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:34.531365 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:34.532010 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:34.532016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:34.532019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:34.532808 (+ 789us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:34.533748 (+ 940us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:34.534447 (+ 699us) server_negotiation.cc:299] Negotiation successful
0927 21:26:34.534889 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":751,"thread_start_us":413,"threads_started":1}
I20260927 21:26:34.539525 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P b88d86270ab84a839fa55a111280df46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b88d86270ab84a839fa55a111280df46. Latest consensus state: current_term: 1 leader_uuid: "b88d86270ab84a839fa55a111280df46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88d86270ab84a839fa55a111280df46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44593 } } }
I20260927 21:26:34.539914 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P b88d86270ab84a839fa55a111280df46 [sys.catalog]: This master's current role is: LEADER
I20260927 21:26:34.539638 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P b88d86270ab84a839fa55a111280df46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b88d86270ab84a839fa55a111280df46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b88d86270ab84a839fa55a111280df46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44593 } } }
I20260927 21:26:34.540689 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P b88d86270ab84a839fa55a111280df46 [sys.catalog]: This master's current role is: LEADER
I20260927 21:26:34.544138 488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:26:34.549499 488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:26:34.559072 488 catalog_manager.cc:1347] Generated new cluster ID: 83de207b0bb24c4f86cf22b4d93e2f54
I20260927 21:26:34.559207 488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:26:34.585650 488 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:26:34.588091 488 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:26:34.602202 488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b88d86270ab84a839fa55a111280df46: Generated new TSK 0
I20260927 21:26:34.603379 488 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 21:26:34.658499 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44593
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:26:35.007812 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:26:35.008131 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:26:35.008229 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:26:35.016075 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:26:35.016201 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:26:35.016317 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:26:35.026340 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44593
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:26:35.027691 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:26:35.029000 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:26:35.043814 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:35.043785 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:35.044373 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:26:35.045310 501 server_base.cc:1034] running on GCE node
I20260927 21:26:35.046030 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:26:35.047076 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:26:35.048290 501 hybrid_clock.cc:648] HybridClock initialized: now 1790544395048259 us; error 44 us; skew 500 ppm
I20260927 21:26:35.048636 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:26:35 dist-test-slave-bxm0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:26:35.052090 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 21:26:35.053823 501 webserver.cc:460] Webserver started at http://127.0.97.1:33649/ using document root <none> and password file <none>
I20260927 21:26:35.054572 501 fs_manager.cc:359] Metadata directory not provided
I20260927 21:26:35.054708 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:26:35.055042 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:26:35.058740 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-0/data/instance:
uuid: "23870250f7b74ae2831357d247a92251"
format_stamp: "Formatted at 2026-09-27 21:26:35 on dist-test-slave-bxm0"
server_key: "13504529c8b7501abcd00028bc7d72a9"
server_key_iv: "71d7d0ade91495bda66fc3ce5159e8c3"
server_key_version: "encryptionkey@0"
I20260927 21:26:35.059592 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-0/wal/instance:
uuid: "23870250f7b74ae2831357d247a92251"
format_stamp: "Formatted at 2026-09-27 21:26:35 on dist-test-slave-bxm0"
server_key: "13504529c8b7501abcd00028bc7d72a9"
server_key_iv: "71d7d0ade91495bda66fc3ce5159e8c3"
server_key_version: "encryptionkey@0"
I20260927 21:26:35.065217 501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 21:26:35.069031 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:35.070547 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:26:35.070749 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "23870250f7b74ae2831357d247a92251"
format_stamp: "Formatted at 2026-09-27 21:26:35 on dist-test-slave-bxm0"
server_key: "13504529c8b7501abcd00028bc7d72a9"
server_key_iv: "71d7d0ade91495bda66fc3ce5159e8c3"
server_key_version: "encryptionkey@0"
I20260927 21:26:35.070950 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:26:35.087930 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:26:35.093020 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:26:35.093283 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:26:35.094347 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:26:35.095703 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:26:35.095780 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:35.095868 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:26:35.095904 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:35.132200 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34277
I20260927 21:26:35.132283 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34277 every 8 connection(s)
I20260927 21:26:35.133533 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:26:35.136264 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 501
I20260927 21:26:35.136480 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-0/wal/instance
I20260927 21:26:35.137089 388 external_mini_cluster.cc:1428] Setting key 397a6f03e29d7a3096fa2a0296575883
Sep 27 21:26:35 dist-test-slave-bxm0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 21:26:35.157634 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:35.136208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58335 (local address 127.0.97.62:44593)
0927 21:26:35.138881 (+ 2673us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:35.138886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:35.138904 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:26:35.140300 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:35.140308 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:35.140408 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:35.140533 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:35.142127 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:35.143314 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:35.146548 (+ 3234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:35.146964 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:35.147075 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:35.151591 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:35.151596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:35.151600 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:35.151635 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:35.154367 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:35.155243 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:35.155254 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:35.155256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:35.155372 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:35.155829 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:35.155834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:35.155839 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:35.156103 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:35.156261 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:35.157006 (+ 745us) server_negotiation.cc:299] Negotiation successful
0927 21:26:35.157236 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2561,"thread_start_us":2459,"threads_started":1}
I20260927 21:26:35.158663 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:35.136966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44593 (local address 127.0.97.1:58335)
0927 21:26:35.137976 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:35.138029 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:35.140004 (+ 1975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:35.140743 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:35.140751 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:35.141295 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:26:35.141945 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:35.141962 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:35.144009 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:35.144017 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:35.146373 (+ 2356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:35.146394 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:35.147223 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:35.147233 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:35.147598 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:35.148424 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:26:35.148461 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:26:35.151328 (+ 2867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:26:35.154535 (+ 3207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:35.154545 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:35.154565 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:35.155087 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:35.155490 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:35.155495 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:35.155498 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:35.155704 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:35.156239 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:26:35.156248 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:26:35.156629 (+ 381us) client_negotiation.cc:769] Sending connection context
0927 21:26:35.156910 (+ 281us) client_negotiation.cc:240] Negotiation successful
0927 21:26:35.157478 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":382,"threads_started":1}
I20260927 21:26:35.160382 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44593
I20260927 21:26:35.160727 629 heartbeater.cc:464] Registering TS with master...
I20260927 21:26:35.161489 629 heartbeater.cc:511] Master 127.0.97.62:44593 requested a full tablet report, sending...
I20260927 21:26:35.163666 440 ts_manager.cc:194] Registered new tserver with Master: 23870250f7b74ae2831357d247a92251 (127.0.97.1:34277)
I20260927 21:26:35.165122 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 21:26:35.181483 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44593
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:26:35.528851 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:26:35.529142 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:26:35.529237 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:26:35.536870 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:26:35.536995 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:26:35.537192 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:26:35.548128 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44593
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:26:35.549399 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:26:35.550822 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:26:35.564730 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:35.564730 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:35.564881 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:26:35.566073 635 server_base.cc:1034] running on GCE node
I20260927 21:26:35.566766 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:26:35.567819 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:26:35.569042 635 hybrid_clock.cc:648] HybridClock initialized: now 1790544395568983 us; error 76 us; skew 500 ppm
I20260927 21:26:35.569382 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:26:35 dist-test-slave-bxm0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:26:35.573127 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 21:26:35.574987 635 webserver.cc:460] Webserver started at http://127.0.97.2:40935/ using document root <none> and password file <none>
I20260927 21:26:35.575881 635 fs_manager.cc:359] Metadata directory not provided
I20260927 21:26:35.575994 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:26:35.576393 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:26:35.579855 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-1/data/instance:
uuid: "d6b3cc0ece55472da36e4d89d319010c"
format_stamp: "Formatted at 2026-09-27 21:26:35 on dist-test-slave-bxm0"
server_key: "0c04ee8aabf03683f056c9fe3f311c86"
server_key_iv: "55823295b4dcd3b56088a38fd6cc52d4"
server_key_version: "encryptionkey@0"
I20260927 21:26:35.580724 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d6b3cc0ece55472da36e4d89d319010c"
format_stamp: "Formatted at 2026-09-27 21:26:35 on dist-test-slave-bxm0"
server_key: "0c04ee8aabf03683f056c9fe3f311c86"
server_key_iv: "55823295b4dcd3b56088a38fd6cc52d4"
server_key_version: "encryptionkey@0"
I20260927 21:26:35.586599 635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260927 21:26:35.590553 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:35.592083 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:26:35.592267 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "d6b3cc0ece55472da36e4d89d319010c"
format_stamp: "Formatted at 2026-09-27 21:26:35 on dist-test-slave-bxm0"
server_key: "0c04ee8aabf03683f056c9fe3f311c86"
server_key_iv: "55823295b4dcd3b56088a38fd6cc52d4"
server_key_version: "encryptionkey@0"
I20260927 21:26:35.592494 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:26:35.618026 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:26:35.623899 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:26:35.624337 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:26:35.625990 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:26:35.628015 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:26:35.628122 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:35.628271 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:26:35.628336 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:35.668237 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37617
I20260927 21:26:35.668370 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37617 every 8 connection(s)
I20260927 21:26:35.669704 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:26:35.670068 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 635
I20260927 21:26:35.670290 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-1/wal/instance
I20260927 21:26:35.671353 388 external_mini_cluster.cc:1428] Setting key 262ec4a081da1ca9da7ce3d4151b36ac
Sep 27 21:26:35 dist-test-slave-bxm0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 21:26:35.697886 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:35.673164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39919 (local address 127.0.97.62:44593)
0927 21:26:35.673690 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:35.673698 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:35.675104 (+ 1406us) server_negotiation.cc:407] Connection header received
0927 21:26:35.678708 (+ 3604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:35.678713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:35.678796 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:35.678900 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:35.680942 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:35.681750 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:35.684597 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:35.685072 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:35.685194 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:35.690057 (+ 4863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:35.690063 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:35.690066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:35.690102 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:35.693403 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:35.694642 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:35.694647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:35.694649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:35.694783 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:35.695380 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:35.695384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:35.695386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:35.695648 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:35.695755 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:35.697232 (+ 1477us) server_negotiation.cc:299] Negotiation successful
0927 21:26:35.697409 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":285,"threads_started":1}
I20260927 21:26:35.699481 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:35.674002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44593 (local address 127.0.97.2:39919)
0927 21:26:35.674906 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:35.674985 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:35.678329 (+ 3344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:35.679269 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:35.679282 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:35.679886 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:26:35.680721 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:35.680749 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:35.681924 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:35.681934 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:35.684367 (+ 2433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:35.684390 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:35.685368 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:35.685379 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:35.685793 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:35.686681 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:26:35.686725 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:26:35.689776 (+ 3051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:26:35.693650 (+ 3874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:35.693664 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:35.693710 (+ 46us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:35.694461 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:35.694919 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:35.694930 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:35.694937 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:35.695249 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:35.696175 (+ 926us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:26:35.696188 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:26:35.696738 (+ 550us) client_negotiation.cc:769] Sending connection context
0927 21:26:35.697100 (+ 362us) client_negotiation.cc:240] Negotiation successful
0927 21:26:35.697779 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":336,"threads_started":1}
I20260927 21:26:35.701155 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44593
I20260927 21:26:35.701505 763 heartbeater.cc:464] Registering TS with master...
I20260927 21:26:35.702287 763 heartbeater.cc:511] Master 127.0.97.62:44593 requested a full tablet report, sending...
I20260927 21:26:35.703780 440 ts_manager.cc:194] Registered new tserver with Master: d6b3cc0ece55472da36e4d89d319010c (127.0.97.2:37617)
I20260927 21:26:35.704638 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 21:26:35.717585 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44593
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:26:36.067225 769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:26:36.067544 769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:26:36.067660 769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:26:36.075335 769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:26:36.075460 769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:26:36.075639 769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:26:36.085980 769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44593
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:26:36.087752 769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:26:36.089094 769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:26:36.103868 774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:36.103847 775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:36.103799 777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:26:36.105726 769 server_base.cc:1034] running on GCE node
I20260927 21:26:36.106351 769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:26:36.107321 769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:26:36.108563 769 hybrid_clock.cc:648] HybridClock initialized: now 1790544396108508 us; error 53 us; skew 500 ppm
I20260927 21:26:36.108904 769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:26:36 dist-test-slave-bxm0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:26:36.112093 769 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 21:26:36.113732 769 webserver.cc:460] Webserver started at http://127.0.97.3:33585/ using document root <none> and password file <none>
I20260927 21:26:36.114456 769 fs_manager.cc:359] Metadata directory not provided
I20260927 21:26:36.114609 769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:26:36.114920 769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:26:36.117916 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-2/data/instance:
uuid: "771ed5b5668d40f7b0a8a965a463785d"
format_stamp: "Formatted at 2026-09-27 21:26:36 on dist-test-slave-bxm0"
server_key: "7dda6b3299f863fa2e84a1a025378dc6"
server_key_iv: "48e7d3f53bf6da0474add2f55b4d7418"
server_key_version: "encryptionkey@0"
I20260927 21:26:36.118741 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-2/wal/instance:
uuid: "771ed5b5668d40f7b0a8a965a463785d"
format_stamp: "Formatted at 2026-09-27 21:26:36 on dist-test-slave-bxm0"
server_key: "7dda6b3299f863fa2e84a1a025378dc6"
server_key_iv: "48e7d3f53bf6da0474add2f55b4d7418"
server_key_version: "encryptionkey@0"
I20260927 21:26:36.124055 769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:26:36.127719 784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:36.129161 769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 21:26:36.129354 769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "771ed5b5668d40f7b0a8a965a463785d"
format_stamp: "Formatted at 2026-09-27 21:26:36 on dist-test-slave-bxm0"
server_key: "7dda6b3299f863fa2e84a1a025378dc6"
server_key_iv: "48e7d3f53bf6da0474add2f55b4d7418"
server_key_version: "encryptionkey@0"
I20260927 21:26:36.129580 769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:26:36.157203 769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:26:36.161693 769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:26:36.162019 769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:26:36.163059 769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:26:36.164392 769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:26:36.164503 769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:36.164665 769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:26:36.164742 769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:36.168251 629 heartbeater.cc:503] Master 127.0.97.62:44593 was elected leader, sending a full tablet report...
I20260927 21:26:36.202132 769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46599
I20260927 21:26:36.202174 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46599 every 8 connection(s)
I20260927 21:26:36.203454 769 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:26:36.206657 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 769
I20260927 21:26:36.206851 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790544390748963-388-0/minicluster-data/ts-2/wal/instance
I20260927 21:26:36.207295 388 external_mini_cluster.cc:1428] Setting key 57f04118b3d249d004ae8b8a0f1da7ec
Sep 27 21:26:36 dist-test-slave-bxm0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:26:36.224937 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:36.206068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39665 (local address 127.0.97.62:44593)
0927 21:26:36.206612 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:36.206620 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:36.207718 (+ 1098us) server_negotiation.cc:407] Connection header received
0927 21:26:36.209951 (+ 2233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:36.209957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:36.210081 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:36.210214 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:36.211876 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:36.212715 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:36.214419 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:36.214841 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:36.214941 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:36.218870 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:36.218875 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:36.218878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:36.218912 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:36.221421 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:36.222262 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:36.222267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:36.222269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:36.222394 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:36.222939 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:36.222944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:36.222947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:36.223330 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:36.223488 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:36.224321 (+ 833us) server_negotiation.cc:299] Negotiation successful
0927 21:26:36.224553 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":249,"threads_started":1}
I20260927 21:26:36.226071 900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:36.206652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44593 (local address 127.0.97.3:39665)
0927 21:26:36.207532 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:36.207576 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:36.209568 (+ 1992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:36.210441 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:36.210451 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:36.211101 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:26:36.211695 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:36.211714 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:36.212876 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:36.212880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:36.214279 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:36.214295 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:36.215043 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:36.215047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:36.215282 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:36.216375 (+ 1093us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:26:36.216398 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:26:36.218649 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:26:36.221566 (+ 2917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:36.221574 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:36.221589 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:36.222083 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:36.222526 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:36.222531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:36.222534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:36.222799 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:36.223462 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:26:36.223471 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:26:36.223937 (+ 466us) client_negotiation.cc:769] Sending connection context
0927 21:26:36.224220 (+ 283us) client_negotiation.cc:240] Negotiation successful
0927 21:26:36.224791 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":387,"threads_started":1}
I20260927 21:26:36.227559 897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44593
I20260927 21:26:36.227862 897 heartbeater.cc:464] Registering TS with master...
I20260927 21:26:36.228556 897 heartbeater.cc:511] Master 127.0.97.62:44593 requested a full tablet report, sending...
I20260927 21:26:36.229918 440 ts_manager.cc:194] Registered new tserver with Master: 771ed5b5668d40f7b0a8a965a463785d (127.0.97.3:46599)
I20260927 21:26:36.230686 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39665
I20260927 21:26:36.236403 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 21:26:36 dist-test-slave-bxm0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544396, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:26:36 dist-test-slave-bxm0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544396, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 21:26:36.281854 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:36.265382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44416 (local address 127.0.97.1:34277)
0927 21:26:36.266056 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:36.266064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:36.266095 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:26:36.266157 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:36.266164 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:36.266374 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:36.266506 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:36.268249 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:36.269257 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:36.270671 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:36.271251 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:36.271366 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:36.275584 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:36.275600 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:36.275615 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:36.275710 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:36.278632 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:36.279409 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:36.279416 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:36.279418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:36.279554 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:36.279992 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:36.279996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:36.279999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:36.280374 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:36.280617 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:36.281132 (+ 515us) server_negotiation.cc:299] Negotiation successful
0927 21:26:36.281348 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":382,"threads_started":1}
W20260927 21:26:36.283327 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44416
Sep 27 21:26:36 dist-test-slave-bxm0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:26:36 dist-test-slave-bxm0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 21:26:36.328953 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:36.314924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44424 (local address 127.0.97.1:34277)
0927 21:26:36.315159 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:36.315167 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:36.315476 (+ 309us) server_negotiation.cc:407] Connection header received
0927 21:26:36.315704 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:36.315708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:36.315771 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:36.315878 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:36.317203 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:36.317982 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:36.319201 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:36.319616 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:36.319753 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:36.323790 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:36.323795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:36.323799 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:36.323834 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:36.326425 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:36.327093 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:36.327099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:36.327103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:36.327231 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:36.327591 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:36.327597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:36.327600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:36.327855 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:36.328093 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:36.328401 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 21:26:36.328631 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 21:26:36.330668 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 501
I20260927 21:26:36.344851 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:36.331790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44438 (local address 127.0.97.1:34277)
0927 21:26:36.332006 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:36.332012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:36.332221 (+ 209us) server_negotiation.cc:407] Connection header received
0927 21:26:36.332416 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:36.332421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:36.332488 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:36.332604 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:36.333961 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:36.334792 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:36.336131 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:36.336536 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:36.336667 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:36.339036 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:36.339042 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:36.339054 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:36.339084 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:36.341671 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:36.342814 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:36.342820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:36.342823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:36.343007 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:36.343418 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:36.343423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:36.343426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:36.343700 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:36.343950 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:36.344260 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 21:26:36.344473 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 21:26:36.345712 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:26:36.428848 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 501
I20260927 21:26:36.445583 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 635
Sep 27 21:26:36 dist-test-slave-bxm0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 21:26:36.462226 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:36.446571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43832 (local address 127.0.97.2:37617)
0927 21:26:36.447100 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:36.447108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:36.447135 (+ 27us) server_negotiation.cc:407] Connection header received
0927 21:26:36.447203 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:36.447210 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:36.447419 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:36.447555 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:36.448741 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:36.449869 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:36.451092 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:36.451795 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:36.451910 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:36.456488 (+ 4578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:36.456509 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:36.456524 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:36.456562 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:36.459437 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:36.460101 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:36.460108 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:36.460111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:36.460230 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:36.460560 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:36.460563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:36.460565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:36.460900 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:36.461121 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:36.461445 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 21:26:36.461813 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":289,"threads_started":1}
I20260927 21:26:36.463471 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:26:36.545857 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 635
I20260927 21:26:36.568080 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 769
Sep 27 21:26:36 dist-test-slave-bxm0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 21:26:36.588689 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:36.569144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34170 (local address 127.0.97.3:46599)
0927 21:26:36.569828 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:36.569835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:36.569864 (+ 29us) server_negotiation.cc:407] Connection header received
0927 21:26:36.569922 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:36.569928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:36.570145 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:36.570273 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:36.571454 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:36.572520 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:36.573852 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:36.574426 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:36.574534 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:36.582940 (+ 8406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:36.582956 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:36.582974 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:36.583025 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:36.586050 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:36.586788 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:36.586794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:36.586797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:36.586919 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:36.587305 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:36.587308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:36.587311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:36.587674 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:36.587907 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:36.588211 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 21:26:36.588410 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":331,"threads_started":1}
I20260927 21:26:36.590108 892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:26:36.673117 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 769
I20260927 21:26:36.689496 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 408
I20260927 21:26:36.690727 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:26:36.794232 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 408
2026-09-27T21:26:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (6002 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:26:36 dist-test-slave-bxm0 krb5kdc[919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:26:36 dist-test-slave-bxm0 krb5kdc[919](info): set up 2 sockets
Sep 27 21:26:36 dist-test-slave-bxm0 krb5kdc[919](info): commencing operation
krb5kdc: starting...
W20260927 21:26:37.873579 388 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:26:37 dist-test-slave-bxm0 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:26:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:26:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:26:38.060583 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35603
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35603
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:26:38.394809 932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:26:38.395115 932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:26:38.395269 932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:26:38.403126 932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:26:38.403264 932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:26:38.403352 932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:26:38.403455 932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:26:38.403546 932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:26:38.413780 932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:26:38.415181 932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:26:38.416497 932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:26:38.428337 937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:38.428511 938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:38.428582 940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:26:38.429339 932 server_base.cc:1034] running on GCE node
I20260927 21:26:38.430258 932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:26:38.431838 932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:26:38.433100 932 hybrid_clock.cc:648] HybridClock initialized: now 1790544398433062 us; error 58 us; skew 500 ppm
I20260927 21:26:38.433486 932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:26:38 dist-test-slave-bxm0 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:26:38.436856 932 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:26:38.438575 932 webserver.cc:460] Webserver started at http://127.0.97.62:41565/ using document root <none> and password file <none>
I20260927 21:26:38.439348 932 fs_manager.cc:359] Metadata directory not provided
I20260927 21:26:38.439512 932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:26:38.439853 932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:26:38.443186 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "17f4251e5e3b436081235ad88d01045e"
format_stamp: "Formatted at 2026-09-27 21:26:38 on dist-test-slave-bxm0"
server_key: "e36f84085f44e12a82a4542f11b0d577"
server_key_iv: "d5b989e0390420c86619aef065721173"
server_key_version: "encryptionkey@0"
I20260927 21:26:38.444023 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "17f4251e5e3b436081235ad88d01045e"
format_stamp: "Formatted at 2026-09-27 21:26:38 on dist-test-slave-bxm0"
server_key: "e36f84085f44e12a82a4542f11b0d577"
server_key_iv: "d5b989e0390420c86619aef065721173"
server_key_version: "encryptionkey@0"
I20260927 21:26:38.450011 932 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260927 21:26:38.453831 947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:38.455307 932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:26:38.455499 932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "17f4251e5e3b436081235ad88d01045e"
format_stamp: "Formatted at 2026-09-27 21:26:38 on dist-test-slave-bxm0"
server_key: "e36f84085f44e12a82a4542f11b0d577"
server_key_iv: "d5b989e0390420c86619aef065721173"
server_key_version: "encryptionkey@0"
I20260927 21:26:38.455709 932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:26:38.480137 932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:26:38.484954 932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:26:38.485275 932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:26:38.504298 932 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35603
I20260927 21:26:38.504318 998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35603 every 8 connection(s)
I20260927 21:26:38.505820 932 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:26:38.507877 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 932
I20260927 21:26:38.508112 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:26:38.508927 388 external_mini_cluster.cc:1428] Setting key c945ae22756ecb00a88e7e053b9aff5d
I20260927 21:26:38.511128 999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 21:26:38 dist-test-slave-bxm0 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:26:38.525056 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17f4251e5e3b436081235ad88d01045e: Bootstrap starting.
I20260927 21:26:38.529456 999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17f4251e5e3b436081235ad88d01045e: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:38.530763 999 log.cc:824] T 00000000000000000000000000000000 P 17f4251e5e3b436081235ad88d01045e: Log is configured to *not* fsync() on all Append() calls
I20260927 21:26:38.534214 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:38.511088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59046 (local address 127.0.97.62:35603)
0927 21:26:38.511958 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:38.511970 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:38.512011 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:26:38.513737 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:38.513761 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:38.514188 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:38.514538 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:38.516282 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:38.517712 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:38.519444 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:38.520165 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:38.520313 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:38.525080 (+ 4767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:38.525095 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:38.525117 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:38.525167 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:38.529243 (+ 4076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:38.530104 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:38.530112 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:38.530121 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:38.530321 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:38.530683 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:38.530688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:38.530692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:38.531466 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:38.531720 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:38.532374 (+ 654us) server_negotiation.cc:299] Negotiation successful
0927 21:26:38.532844 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":408,"threads_started":1}
I20260927 21:26:38.534379 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17f4251e5e3b436081235ad88d01045e: No bootstrap required, opened a new log
I20260927 21:26:38.542728 999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17f4251e5e3b436081235ad88d01045e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f4251e5e3b436081235ad88d01045e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35603 } }
I20260927 21:26:38.543125 999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17f4251e5e3b436081235ad88d01045e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:38.543203 999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17f4251e5e3b436081235ad88d01045e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17f4251e5e3b436081235ad88d01045e, State: Initialized, Role: FOLLOWER
I20260927 21:26:38.543797 999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17f4251e5e3b436081235ad88d01045e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f4251e5e3b436081235ad88d01045e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35603 } }
I20260927 21:26:38.544008 999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17f4251e5e3b436081235ad88d01045e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:26:38.544117 999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17f4251e5e3b436081235ad88d01045e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:26:38.544245 999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17f4251e5e3b436081235ad88d01045e [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:38.546648 999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17f4251e5e3b436081235ad88d01045e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f4251e5e3b436081235ad88d01045e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35603 } }
I20260927 21:26:38.547142 999 leader_election.cc:302] T 00000000000000000000000000000000 P 17f4251e5e3b436081235ad88d01045e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17f4251e5e3b436081235ad88d01045e; no voters:
I20260927 21:26:38.547705 999 leader_election.cc:288] T 00000000000000000000000000000000 P 17f4251e5e3b436081235ad88d01045e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:26:38.547863 1004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17f4251e5e3b436081235ad88d01045e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:26:38.548566 1004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17f4251e5e3b436081235ad88d01045e [term 1 LEADER]: Becoming Leader. State: Replica: 17f4251e5e3b436081235ad88d01045e, State: Running, Role: LEADER
I20260927 21:26:38.549194 1004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17f4251e5e3b436081235ad88d01045e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f4251e5e3b436081235ad88d01045e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35603 } }
I20260927 21:26:38.549937 999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17f4251e5e3b436081235ad88d01045e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:26:38.552171 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17f4251e5e3b436081235ad88d01045e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17f4251e5e3b436081235ad88d01045e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f4251e5e3b436081235ad88d01045e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35603 } } }
I20260927 21:26:38.552639 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17f4251e5e3b436081235ad88d01045e [sys.catalog]: This master's current role is: LEADER
I20260927 21:26:38.552224 1006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17f4251e5e3b436081235ad88d01045e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17f4251e5e3b436081235ad88d01045e. Latest consensus state: current_term: 1 leader_uuid: "17f4251e5e3b436081235ad88d01045e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f4251e5e3b436081235ad88d01045e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35603 } } }
I20260927 21:26:38.552860 1006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17f4251e5e3b436081235ad88d01045e [sys.catalog]: This master's current role is: LEADER
I20260927 21:26:38.555253 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:26:38.561192 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:26:38.569859 1010 catalog_manager.cc:1347] Generated new cluster ID: e89b8e7f647047e9be747767a9b3970e
I20260927 21:26:38.570022 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:26:38.597434 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:26:38.599531 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:26:38.624372 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17f4251e5e3b436081235ad88d01045e: Generated new TSK 0
I20260927 21:26:38.625491 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 21:26:38.677263 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35603
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:26:39.010649 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:26:39.010968 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:26:39.011075 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:26:39.018493 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:26:39.018630 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:26:39.018748 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:26:39.028748 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35603
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:26:39.030087 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:26:39.031410 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:26:39.045676 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:39.045713 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:39.045653 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:26:39.046535 1025 server_base.cc:1034] running on GCE node
I20260927 21:26:39.047149 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:26:39.048079 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:26:39.049276 1025 hybrid_clock.cc:648] HybridClock initialized: now 1790544399049206 us; error 59 us; skew 500 ppm
I20260927 21:26:39.049639 1025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:26:39 dist-test-slave-bxm0 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:26:39.053099 1025 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 21:26:39.054622 1025 webserver.cc:460] Webserver started at http://127.0.97.1:33919/ using document root <none> and password file <none>
I20260927 21:26:39.055336 1025 fs_manager.cc:359] Metadata directory not provided
I20260927 21:26:39.055441 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:26:39.055740 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:26:39.058840 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-0/data/instance:
uuid: "85aa45aa04694932af2607302c8bde36"
format_stamp: "Formatted at 2026-09-27 21:26:39 on dist-test-slave-bxm0"
server_key: "6fa017c59f9bbee3f43e409a1c53976e"
server_key_iv: "6115a3acbd6fe2746f2a22c1cb8a3d19"
server_key_version: "encryptionkey@0"
I20260927 21:26:39.059716 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-0/wal/instance:
uuid: "85aa45aa04694932af2607302c8bde36"
format_stamp: "Formatted at 2026-09-27 21:26:39 on dist-test-slave-bxm0"
server_key: "6fa017c59f9bbee3f43e409a1c53976e"
server_key_iv: "6115a3acbd6fe2746f2a22c1cb8a3d19"
server_key_version: "encryptionkey@0"
I20260927 21:26:39.065169 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 21:26:39.068734 1040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:39.070291 1025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:26:39.070472 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "85aa45aa04694932af2607302c8bde36"
format_stamp: "Formatted at 2026-09-27 21:26:39 on dist-test-slave-bxm0"
server_key: "6fa017c59f9bbee3f43e409a1c53976e"
server_key_iv: "6115a3acbd6fe2746f2a22c1cb8a3d19"
server_key_version: "encryptionkey@0"
I20260927 21:26:39.070680 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:26:39.097896 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:26:39.102311 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:26:39.102636 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:26:39.103665 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:26:39.105044 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:26:39.105159 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:39.105305 1025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:26:39.105381 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:39.138641 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40919
I20260927 21:26:39.138692 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40919 every 8 connection(s)
I20260927 21:26:39.139988 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:26:39.145169 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 1025
I20260927 21:26:39.145394 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-0/wal/instance
I20260927 21:26:39.146174 388 external_mini_cluster.cc:1428] Setting key 458a3defb5b194c9de146ab03679bd44
Sep 27 21:26:39 dist-test-slave-bxm0 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 21:26:39.164361 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:39.142646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37491 (local address 127.0.97.62:35603)
0927 21:26:39.143090 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:39.143094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:39.144026 (+ 932us) server_negotiation.cc:407] Connection header received
0927 21:26:39.146712 (+ 2686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:39.146717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:39.146793 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:39.146886 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:39.148657 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:39.149924 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:39.151879 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:39.152650 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:39.152782 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:39.157545 (+ 4763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:39.157551 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:39.157554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:39.157592 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:39.160534 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:39.161706 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:39.161711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:39.161714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:39.161878 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:39.162444 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:39.162448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:39.162451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:39.162740 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:39.162852 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:39.163889 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0927 21:26:39.164101 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":240,"threads_started":1}
I20260927 21:26:39.165431 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:39.143122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35603 (local address 127.0.97.1:37491)
0927 21:26:39.143848 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:39.143887 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:39.146321 (+ 2434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:39.147181 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:39.147190 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:39.147728 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:26:39.148474 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:39.148494 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:39.150107 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:39.150114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:39.151678 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:39.151701 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:39.152938 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:39.152947 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:39.153315 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:39.154231 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:26:39.154268 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:26:39.157211 (+ 2943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:26:39.160739 (+ 3528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:39.160753 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:39.160781 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:39.161484 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:39.162006 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:39.162012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:39.162016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:39.162299 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:39.162911 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:26:39.162924 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:26:39.163421 (+ 497us) client_negotiation.cc:769] Sending connection context
0927 21:26:39.163786 (+ 365us) client_negotiation.cc:240] Negotiation successful
0927 21:26:39.164220 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":285,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 21:26:39.167203 1153 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35603
I20260927 21:26:39.167567 1153 heartbeater.cc:464] Registering TS with master...
I20260927 21:26:39.168530 1153 heartbeater.cc:511] Master 127.0.97.62:35603 requested a full tablet report, sending...
I20260927 21:26:39.171002 964 ts_manager.cc:194] Registered new tserver with Master: 85aa45aa04694932af2607302c8bde36 (127.0.97.1:40919)
I20260927 21:26:39.172480 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37491
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 21:26:39.188786 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35603
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:26:39.533056 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:26:39.533439 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:26:39.533667 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:26:39.541850 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:26:39.542035 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:26:39.542222 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:26:39.552538 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35603
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:26:39.554001 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:26:39.555399 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:26:39.569262 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:39.569207 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:39.569281 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:26:39.570169 1159 server_base.cc:1034] running on GCE node
I20260927 21:26:39.570762 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:26:39.571772 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:26:39.572999 1159 hybrid_clock.cc:648] HybridClock initialized: now 1790544399572921 us; error 62 us; skew 500 ppm
I20260927 21:26:39.573390 1159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:26:39 dist-test-slave-bxm0 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:26:39.576967 1159 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 21:26:39.578488 1159 webserver.cc:460] Webserver started at http://127.0.97.2:45699/ using document root <none> and password file <none>
I20260927 21:26:39.579164 1159 fs_manager.cc:359] Metadata directory not provided
I20260927 21:26:39.579265 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:26:39.579531 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:26:39.582520 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-1/data/instance:
uuid: "a3c4ab3785dc449f83860d5d51b860ae"
format_stamp: "Formatted at 2026-09-27 21:26:39 on dist-test-slave-bxm0"
server_key: "2bfddf37a1f6993980f64f49da5a878e"
server_key_iv: "7612ac32a141a55b2838a7136c6609da"
server_key_version: "encryptionkey@0"
I20260927 21:26:39.583297 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a3c4ab3785dc449f83860d5d51b860ae"
format_stamp: "Formatted at 2026-09-27 21:26:39 on dist-test-slave-bxm0"
server_key: "2bfddf37a1f6993980f64f49da5a878e"
server_key_iv: "7612ac32a141a55b2838a7136c6609da"
server_key_version: "encryptionkey@0"
I20260927 21:26:39.588681 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260927 21:26:39.592415 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:39.593979 1159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:26:39.594169 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "a3c4ab3785dc449f83860d5d51b860ae"
format_stamp: "Formatted at 2026-09-27 21:26:39 on dist-test-slave-bxm0"
server_key: "2bfddf37a1f6993980f64f49da5a878e"
server_key_iv: "7612ac32a141a55b2838a7136c6609da"
server_key_version: "encryptionkey@0"
I20260927 21:26:39.594384 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:26:39.614315 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:26:39.619053 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:26:39.619370 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:26:39.620389 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:26:39.622164 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:26:39.622274 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:39.622411 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:26:39.622478 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:39.658464 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42913
I20260927 21:26:39.658511 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42913 every 8 connection(s)
I20260927 21:26:39.659894 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:26:39.666695 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 1159
I20260927 21:26:39.666893 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-1/wal/instance
I20260927 21:26:39.667335 388 external_mini_cluster.cc:1428] Setting key 01d7f51d8bdcb313aadc6563f070ada4
Sep 27 21:26:39 dist-test-slave-bxm0 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:26:39.683472 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:39.662537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43309 (local address 127.0.97.62:35603)
0927 21:26:39.662729 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:39.662734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:39.663965 (+ 1231us) server_negotiation.cc:407] Connection header received
0927 21:26:39.665887 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:39.665893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:39.665988 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:39.666102 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:39.667925 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:39.668769 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:39.671235 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:39.671941 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:39.672083 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:39.676781 (+ 4698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:39.676787 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:39.676790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:39.676827 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:39.679502 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:39.680601 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:39.680606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:39.680608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:39.680745 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:39.681337 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:39.681341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:39.681343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:39.681711 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:39.681904 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:39.683021 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0927 21:26:39.683220 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 21:26:39.685127 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:39.663035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35603 (local address 127.0.97.2:43309)
0927 21:26:39.663778 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:39.663817 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:39.665551 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:39.666385 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:39.666395 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:39.667048 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:26:39.667720 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:39.667739 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:39.668945 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:39.668949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:39.671045 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:39.671065 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:39.672217 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:39.672227 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:39.672594 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:39.673453 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:26:39.673495 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:26:39.676489 (+ 2994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:26:39.679675 (+ 3186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:39.679689 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:39.679714 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:39.680451 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:39.680862 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:39.680869 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:39.680876 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:39.681167 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:39.681940 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:26:39.681953 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:26:39.682485 (+ 532us) client_negotiation.cc:769] Sending connection context
0927 21:26:39.682880 (+ 395us) client_negotiation.cc:240] Negotiation successful
0927 21:26:39.683526 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":301,"threads_started":1}
I20260927 21:26:39.686823 1287 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35603
I20260927 21:26:39.687178 1287 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 21:26:39.688123 1287 heartbeater.cc:511] Master 127.0.97.62:35603 requested a full tablet report, sending...
I20260927 21:26:39.690016 964 ts_manager.cc:194] Registered new tserver with Master: a3c4ab3785dc449f83860d5d51b860ae (127.0.97.2:42913)
I20260927 21:26:39.690860 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 21:26:39.709508 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35603
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:26:40.044644 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:26:40.044996 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:26:40.045152 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:26:40.052672 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:26:40.052855 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:26:40.053027 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:26:40.063301 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35603
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:26:40.064690 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:26:40.066100 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:26:40.080065 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:40.080104 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:40.080065 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:26:40.086315 1292 server_base.cc:1034] running on GCE node
I20260927 21:26:40.086982 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:26:40.087990 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:26:40.089227 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790544400089183 us; error 52 us; skew 500 ppm
I20260927 21:26:40.089570 1292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:26:40 dist-test-slave-bxm0 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:26:40.093005 1292 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 21:26:40.094594 1292 webserver.cc:460] Webserver started at http://127.0.97.3:45555/ using document root <none> and password file <none>
I20260927 21:26:40.095372 1292 fs_manager.cc:359] Metadata directory not provided
I20260927 21:26:40.095511 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:26:40.095832 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:26:40.098943 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-2/data/instance:
uuid: "1fda72425b8941788641e15a3b80c068"
format_stamp: "Formatted at 2026-09-27 21:26:40 on dist-test-slave-bxm0"
server_key: "f3f46f4976e86c2a8bac2f9e6d09cfbc"
server_key_iv: "d7eb74b3950df31b59455b1041c6fa0a"
server_key_version: "encryptionkey@0"
I20260927 21:26:40.099783 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1fda72425b8941788641e15a3b80c068"
format_stamp: "Formatted at 2026-09-27 21:26:40 on dist-test-slave-bxm0"
server_key: "f3f46f4976e86c2a8bac2f9e6d09cfbc"
server_key_iv: "d7eb74b3950df31b59455b1041c6fa0a"
server_key_version: "encryptionkey@0"
I20260927 21:26:40.105015 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 21:26:40.108655 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:40.110121 1292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:26:40.110294 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "1fda72425b8941788641e15a3b80c068"
format_stamp: "Formatted at 2026-09-27 21:26:40 on dist-test-slave-bxm0"
server_key: "f3f46f4976e86c2a8bac2f9e6d09cfbc"
server_key_iv: "d7eb74b3950df31b59455b1041c6fa0a"
server_key_version: "encryptionkey@0"
I20260927 21:26:40.110503 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:26:40.130118 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:26:40.134328 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:26:40.134658 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:26:40.135684 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:26:40.137125 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:26:40.137221 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:40.137368 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:26:40.137436 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:40.172587 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46683
I20260927 21:26:40.172647 1419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46683 every 8 connection(s)
I20260927 21:26:40.173979 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:26:40.175836 1153 heartbeater.cc:503] Master 127.0.97.62:35603 was elected leader, sending a full tablet report...
I20260927 21:26:40.177158 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 1292
I20260927 21:26:40.177341 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790544390748963-388-0/minicluster-data/ts-2/wal/instance
I20260927 21:26:40.177896 388 external_mini_cluster.cc:1428] Setting key d9de45635cc24600a18605b44723e596
Sep 27 21:26:40 dist-test-slave-bxm0 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:26:40.195734 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:40.176937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35245 (local address 127.0.97.62:35603)
0927 21:26:40.177112 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:40.177118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:40.178550 (+ 1432us) server_negotiation.cc:407] Connection header received
0927 21:26:40.181031 (+ 2481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:40.181036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:40.181117 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:40.181234 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:40.183373 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.184131 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.185785 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.186172 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.186305 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:40.189744 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:40.189749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:40.189752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:40.189783 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:40.192799 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:40.193652 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:40.193662 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:40.193664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:40.193793 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:40.194208 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:40.194211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:40.194213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:40.194467 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:40.194581 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:40.195307 (+ 726us) server_negotiation.cc:299] Negotiation successful
0927 21:26:40.195482 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:26:40.196668 1422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:40.177503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35603 (local address 127.0.97.3:35245)
0927 21:26:40.178361 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:40.178403 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:40.180723 (+ 2320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:40.181480 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:40.181489 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:40.182536 (+ 1047us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:26:40.183204 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:40.183223 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.184283 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.184288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:40.185563 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:40.185575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.186405 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.186412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:40.186658 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:40.187347 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:26:40.187374 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:26:40.189481 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:26:40.192928 (+ 3447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:40.192935 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:40.192951 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:40.193491 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:40.193886 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:40.193890 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:40.193892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:40.194073 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:40.194565 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:26:40.194570 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:26:40.194904 (+ 334us) client_negotiation.cc:769] Sending connection context
0927 21:26:40.195154 (+ 250us) client_negotiation.cc:240] Negotiation successful
0927 21:26:40.195556 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":360,"threads_started":1}
I20260927 21:26:40.198381 1420 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35603
I20260927 21:26:40.198665 1420 heartbeater.cc:464] Registering TS with master...
I20260927 21:26:40.199430 1420 heartbeater.cc:511] Master 127.0.97.62:35603 requested a full tablet report, sending...
I20260927 21:26:40.200799 964 ts_manager.cc:194] Registered new tserver with Master: 1fda72425b8941788641e15a3b80c068 (127.0.97.3:46683)
I20260927 21:26:40.201577 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35245
I20260927 21:26:40.205875 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:26:40.225101 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:40.212347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59050 (local address 127.0.97.62:35603)
0927 21:26:40.212564 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:40.212572 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:40.212952 (+ 380us) server_negotiation.cc:407] Connection header received
0927 21:26:40.213156 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:40.213166 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:40.213231 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:40.213336 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:40.214907 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.215644 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.216882 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.217282 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.217414 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:40.219778 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:40.219783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:40.219786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:40.219826 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:40.222492 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:40.223160 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:40.223167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:40.223171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:40.223294 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:40.223661 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:40.223668 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:40.223672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:40.223993 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:40.224240 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:40.224566 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 21:26:40.224790 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 21:26:40.231732 964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:26:40.239163 964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:26:40.265079 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:40.252605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55364 (local address 127.0.97.3:46683)
0927 21:26:40.256217 (+ 3612us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:40.256274 (+ 57us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:40.256317 (+ 43us) server_negotiation.cc:407] Connection header received
0927 21:26:40.256498 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:40.256512 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:40.256840 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:40.257662 (+ 822us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:26:40.257996 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.259477 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.262518 (+ 3041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.263858 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.263989 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:40.264225 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:40.264538 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 21:26:40.264717 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3387,"thread_start_us":3200,"threads_started":1}
I20260927 21:26:40.265450 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:40.251323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40919 (local address 127.0.0.1:51262)
0927 21:26:40.252841 (+ 1518us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:40.252899 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:40.253116 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:40.258252 (+ 5136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:40.258256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:40.258284 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:26:40.258720 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:40.258732 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.260679 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.260687 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:40.262798 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:40.262817 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.264624 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.264630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:40.264788 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:40.264823 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 21:26:40.264894 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 21:26:40.265017 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1156,"mutex_wait_us":40,"thread_start_us":3716,"threads_started":1}
I20260927 21:26:40.266062 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:40.251973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46683 (local address 127.0.0.1:55364)
0927 21:26:40.255391 (+ 3418us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:40.255415 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:40.255534 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:40.257087 (+ 1553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:40.257095 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:40.257132 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:26:40.257709 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:40.257725 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.259629 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.259634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:40.262361 (+ 2727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:40.262375 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.264041 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.264047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:40.264253 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:40.264304 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 21:26:40.264432 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 21:26:40.265274 (+ 842us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3244,"mutex_wait_us":322,"thread_start_us":812,"threads_started":1}
I20260927 21:26:40.266376 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:40.251619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51262 (local address 127.0.97.1:40919)
0927 21:26:40.257561 (+ 5942us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:40.257570 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:40.257601 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:26:40.257727 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:40.257736 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:40.258020 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:40.258937 (+ 917us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:26:40.259160 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.260559 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.262963 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.264376 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.265579 (+ 1203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:40.265832 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:40.265904 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 21:26:40.266064 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5764,"thread_start_us":5626,"threads_started":1}
I20260927 21:26:40.268023 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:40.257044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42913 (local address 127.0.0.1:34146)
0927 21:26:40.258263 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:40.258285 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:40.258401 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:40.259319 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:40.259323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:40.259349 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:26:40.259792 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:40.259804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.262258 (+ 2454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.262263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:40.263802 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:40.263812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.266299 (+ 2487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.266305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:40.266499 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:40.266536 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 21:26:40.267697 (+ 1161us) client_negotiation.cc:240] Negotiation successful
0927 21:26:40.267796 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1063,"thread_start_us":902,"threads_started":1}
I20260927 21:26:40.268225 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:40.255693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34146 (local address 127.0.97.2:42913)
0927 21:26:40.257030 (+ 1337us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:40.257042 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:40.258530 (+ 1488us) server_negotiation.cc:407] Connection header received
0927 21:26:40.258641 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:40.258652 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:40.258913 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:40.259102 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:26:40.259963 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.261456 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.263952 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.265862 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.265981 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:40.266163 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:40.267772 (+ 1609us) server_negotiation.cc:299] Negotiation successful
0927 21:26:40.267948 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1139,"thread_start_us":1003,"threads_started":1}
I20260927 21:26:40.270877 1088 tablet_service.cc:1467] Processing CreateTablet for tablet 52c72a152d1446bda39e4f47511a8840 (DEFAULT_TABLE table=test-table [id=7a7d160327374416838efbbadd9efa1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:26:40.271486 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 52c72a152d1446bda39e4f47511a8840 (DEFAULT_TABLE table=test-table [id=7a7d160327374416838efbbadd9efa1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:26:40.271719 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 52c72a152d1446bda39e4f47511a8840 (DEFAULT_TABLE table=test-table [id=7a7d160327374416838efbbadd9efa1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:26:40.272495 1088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52c72a152d1446bda39e4f47511a8840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:40.272747 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52c72a152d1446bda39e4f47511a8840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:40.273005 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52c72a152d1446bda39e4f47511a8840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:40.285449 1438 tablet_bootstrap.cc:492] T 52c72a152d1446bda39e4f47511a8840 P 1fda72425b8941788641e15a3b80c068: Bootstrap starting.
I20260927 21:26:40.288159 1439 tablet_bootstrap.cc:492] T 52c72a152d1446bda39e4f47511a8840 P a3c4ab3785dc449f83860d5d51b860ae: Bootstrap starting.
I20260927 21:26:40.289242 1438 tablet_bootstrap.cc:654] T 52c72a152d1446bda39e4f47511a8840 P 1fda72425b8941788641e15a3b80c068: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:40.289316 1440 tablet_bootstrap.cc:492] T 52c72a152d1446bda39e4f47511a8840 P 85aa45aa04694932af2607302c8bde36: Bootstrap starting.
I20260927 21:26:40.290684 1438 log.cc:824] T 52c72a152d1446bda39e4f47511a8840 P 1fda72425b8941788641e15a3b80c068: Log is configured to *not* fsync() on all Append() calls
I20260927 21:26:40.293399 1440 tablet_bootstrap.cc:654] T 52c72a152d1446bda39e4f47511a8840 P 85aa45aa04694932af2607302c8bde36: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:40.293430 1438 tablet_bootstrap.cc:492] T 52c72a152d1446bda39e4f47511a8840 P 1fda72425b8941788641e15a3b80c068: No bootstrap required, opened a new log
I20260927 21:26:40.293767 1438 ts_tablet_manager.cc:1397] T 52c72a152d1446bda39e4f47511a8840 P 1fda72425b8941788641e15a3b80c068: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 21:26:40.294688 1440 log.cc:824] T 52c72a152d1446bda39e4f47511a8840 P 85aa45aa04694932af2607302c8bde36: Log is configured to *not* fsync() on all Append() calls
I20260927 21:26:40.295282 1439 tablet_bootstrap.cc:654] T 52c72a152d1446bda39e4f47511a8840 P a3c4ab3785dc449f83860d5d51b860ae: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:40.296845 1439 log.cc:824] T 52c72a152d1446bda39e4f47511a8840 P a3c4ab3785dc449f83860d5d51b860ae: Log is configured to *not* fsync() on all Append() calls
I20260927 21:26:40.299923 1440 tablet_bootstrap.cc:492] T 52c72a152d1446bda39e4f47511a8840 P 85aa45aa04694932af2607302c8bde36: No bootstrap required, opened a new log
I20260927 21:26:40.300366 1439 tablet_bootstrap.cc:492] T 52c72a152d1446bda39e4f47511a8840 P a3c4ab3785dc449f83860d5d51b860ae: No bootstrap required, opened a new log
I20260927 21:26:40.300652 1439 ts_tablet_manager.cc:1397] T 52c72a152d1446bda39e4f47511a8840 P a3c4ab3785dc449f83860d5d51b860ae: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260927 21:26:40.300647 1440 ts_tablet_manager.cc:1397] T 52c72a152d1446bda39e4f47511a8840 P 85aa45aa04694932af2607302c8bde36: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260927 21:26:40.302181 1438 raft_consensus.cc:348] T 52c72a152d1446bda39e4f47511a8840 P 1fda72425b8941788641e15a3b80c068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aa45aa04694932af2607302c8bde36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40919 } } peers { permanent_uuid: "a3c4ab3785dc449f83860d5d51b860ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42913 } } peers { permanent_uuid: "1fda72425b8941788641e15a3b80c068" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46683 } }
I20260927 21:26:40.302573 1438 raft_consensus.cc:374] T 52c72a152d1446bda39e4f47511a8840 P 1fda72425b8941788641e15a3b80c068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:40.302683 1438 raft_consensus.cc:729] T 52c72a152d1446bda39e4f47511a8840 P 1fda72425b8941788641e15a3b80c068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fda72425b8941788641e15a3b80c068, State: Initialized, Role: FOLLOWER
I20260927 21:26:40.303339 1438 consensus_queue.cc:260] T 52c72a152d1446bda39e4f47511a8840 P 1fda72425b8941788641e15a3b80c068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aa45aa04694932af2607302c8bde36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40919 } } peers { permanent_uuid: "a3c4ab3785dc449f83860d5d51b860ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42913 } } peers { permanent_uuid: "1fda72425b8941788641e15a3b80c068" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46683 } }
I20260927 21:26:40.304517 1420 heartbeater.cc:503] Master 127.0.97.62:35603 was elected leader, sending a full tablet report...
I20260927 21:26:40.305109 1438 ts_tablet_manager.cc:1428] T 52c72a152d1446bda39e4f47511a8840 P 1fda72425b8941788641e15a3b80c068: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260927 21:26:40.311771 1440 raft_consensus.cc:348] T 52c72a152d1446bda39e4f47511a8840 P 85aa45aa04694932af2607302c8bde36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aa45aa04694932af2607302c8bde36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40919 } } peers { permanent_uuid: "a3c4ab3785dc449f83860d5d51b860ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42913 } } peers { permanent_uuid: "1fda72425b8941788641e15a3b80c068" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46683 } }
I20260927 21:26:40.312247 1440 raft_consensus.cc:374] T 52c72a152d1446bda39e4f47511a8840 P 85aa45aa04694932af2607302c8bde36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:40.312358 1440 raft_consensus.cc:729] T 52c72a152d1446bda39e4f47511a8840 P 85aa45aa04694932af2607302c8bde36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85aa45aa04694932af2607302c8bde36, State: Initialized, Role: FOLLOWER
I20260927 21:26:40.313020 1440 consensus_queue.cc:260] T 52c72a152d1446bda39e4f47511a8840 P 85aa45aa04694932af2607302c8bde36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aa45aa04694932af2607302c8bde36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40919 } } peers { permanent_uuid: "a3c4ab3785dc449f83860d5d51b860ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42913 } } peers { permanent_uuid: "1fda72425b8941788641e15a3b80c068" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46683 } }
I20260927 21:26:40.314576 1439 raft_consensus.cc:348] T 52c72a152d1446bda39e4f47511a8840 P a3c4ab3785dc449f83860d5d51b860ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aa45aa04694932af2607302c8bde36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40919 } } peers { permanent_uuid: "a3c4ab3785dc449f83860d5d51b860ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42913 } } peers { permanent_uuid: "1fda72425b8941788641e15a3b80c068" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46683 } }
I20260927 21:26:40.315064 1440 ts_tablet_manager.cc:1428] T 52c72a152d1446bda39e4f47511a8840 P 85aa45aa04694932af2607302c8bde36: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260927 21:26:40.315081 1439 raft_consensus.cc:374] T 52c72a152d1446bda39e4f47511a8840 P a3c4ab3785dc449f83860d5d51b860ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:40.315179 1439 raft_consensus.cc:729] T 52c72a152d1446bda39e4f47511a8840 P a3c4ab3785dc449f83860d5d51b860ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3c4ab3785dc449f83860d5d51b860ae, State: Initialized, Role: FOLLOWER
I20260927 21:26:40.315821 1439 consensus_queue.cc:260] T 52c72a152d1446bda39e4f47511a8840 P a3c4ab3785dc449f83860d5d51b860ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aa45aa04694932af2607302c8bde36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40919 } } peers { permanent_uuid: "a3c4ab3785dc449f83860d5d51b860ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42913 } } peers { permanent_uuid: "1fda72425b8941788641e15a3b80c068" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46683 } }
I20260927 21:26:40.316924 1287 heartbeater.cc:503] Master 127.0.97.62:35603 was elected leader, sending a full tablet report...
I20260927 21:26:40.317585 1439 ts_tablet_manager.cc:1428] T 52c72a152d1446bda39e4f47511a8840 P a3c4ab3785dc449f83860d5d51b860ae: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
W20260927 21:26:40.392303 1154 tablet.cc:2369] T 52c72a152d1446bda39e4f47511a8840 P 85aa45aa04694932af2607302c8bde36: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:26:40.411947 1288 tablet.cc:2369] T 52c72a152d1446bda39e4f47511a8840 P a3c4ab3785dc449f83860d5d51b860ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:26:40.425696 1421 tablet.cc:2369] T 52c72a152d1446bda39e4f47511a8840 P 1fda72425b8941788641e15a3b80c068: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:26:40.668992 1445 raft_consensus.cc:482] T 52c72a152d1446bda39e4f47511a8840 P 85aa45aa04694932af2607302c8bde36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:26:40.669241 1445 raft_consensus.cc:504] T 52c72a152d1446bda39e4f47511a8840 P 85aa45aa04694932af2607302c8bde36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aa45aa04694932af2607302c8bde36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40919 } } peers { permanent_uuid: "a3c4ab3785dc449f83860d5d51b860ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42913 } } peers { permanent_uuid: "1fda72425b8941788641e15a3b80c068" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46683 } }
I20260927 21:26:40.670816 1445 leader_election.cc:288] T 52c72a152d1446bda39e4f47511a8840 P 85aa45aa04694932af2607302c8bde36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3c4ab3785dc449f83860d5d51b860ae (127.0.97.2:42913), 1fda72425b8941788641e15a3b80c068 (127.0.97.3:46683)
I20260927 21:26:40.680534 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:40.671226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53393 (local address 127.0.97.3:46683)
0927 21:26:40.671421 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:40.671427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:40.672133 (+ 706us) server_negotiation.cc:407] Connection header received
0927 21:26:40.672346 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:40.672351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:40.672437 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:40.672553 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:26:40.673375 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.674943 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.677948 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.679223 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.679355 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:40.679462 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:40.680137 (+ 675us) server_negotiation.cc:299] Negotiation successful
0927 21:26:40.680271 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 21:26:40.680549 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:40.671309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42323 (local address 127.0.97.2:42913)
0927 21:26:40.671492 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:40.671500 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:40.672194 (+ 694us) server_negotiation.cc:407] Connection header received
0927 21:26:40.672346 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:40.672351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:40.672437 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:40.672539 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:26:40.673375 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.674927 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.677955 (+ 3028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.679223 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.679361 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:40.679461 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:40.680137 (+ 676us) server_negotiation.cc:299] Negotiation successful
0927 21:26:40.680271 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:26:40.680826 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:40.671207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46683 (local address 127.0.97.1:53393)
0927 21:26:40.672009 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:40.672030 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:40.672206 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:40.672566 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:40.672571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:40.672609 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:26:40.673197 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:40.673209 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.675116 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.675120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:40.677759 (+ 2639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:40.677779 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.679520 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.679525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:40.679835 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:40.680000 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 21:26:40.680103 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 21:26:40.680275 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":328,"spinlock_wait_cycles":384,"thread_start_us":400,"threads_started":1}
I20260927 21:26:40.680847 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:40.671264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42913 (local address 127.0.97.1:42323)
0927 21:26:40.672088 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:40.672103 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:40.672214 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:40.672566 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:40.672571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:40.672616 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:26:40.673199 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:40.673209 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.675108 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.675114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:40.677775 (+ 2661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:40.677783 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.679508 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.679514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:40.679833 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:40.680000 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 21:26:40.680096 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 21:26:40.680275 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":332,"spinlock_wait_cycles":28288,"thread_start_us":436,"threads_started":1}
I20260927 21:26:40.681843 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52c72a152d1446bda39e4f47511a8840" candidate_uuid: "85aa45aa04694932af2607302c8bde36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fda72425b8941788641e15a3b80c068" is_pre_election: true
I20260927 21:26:40.682020 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52c72a152d1446bda39e4f47511a8840" candidate_uuid: "85aa45aa04694932af2607302c8bde36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c4ab3785dc449f83860d5d51b860ae" is_pre_election: true
I20260927 21:26:40.682386 1375 raft_consensus.cc:2454] T 52c72a152d1446bda39e4f47511a8840 P 1fda72425b8941788641e15a3b80c068 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85aa45aa04694932af2607302c8bde36 in term 0.
I20260927 21:26:40.682530 1242 raft_consensus.cc:2454] T 52c72a152d1446bda39e4f47511a8840 P a3c4ab3785dc449f83860d5d51b860ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85aa45aa04694932af2607302c8bde36 in term 0.
I20260927 21:26:40.683008 1044 leader_election.cc:302] T 52c72a152d1446bda39e4f47511a8840 P 85aa45aa04694932af2607302c8bde36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fda72425b8941788641e15a3b80c068, 85aa45aa04694932af2607302c8bde36; no voters:
I20260927 21:26:40.683432 1445 raft_consensus.cc:2781] T 52c72a152d1446bda39e4f47511a8840 P 85aa45aa04694932af2607302c8bde36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:26:40.683580 1445 raft_consensus.cc:482] T 52c72a152d1446bda39e4f47511a8840 P 85aa45aa04694932af2607302c8bde36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:26:40.683729 1445 raft_consensus.cc:3037] T 52c72a152d1446bda39e4f47511a8840 P 85aa45aa04694932af2607302c8bde36 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:40.686022 1445 raft_consensus.cc:504] T 52c72a152d1446bda39e4f47511a8840 P 85aa45aa04694932af2607302c8bde36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aa45aa04694932af2607302c8bde36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40919 } } peers { permanent_uuid: "a3c4ab3785dc449f83860d5d51b860ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42913 } } peers { permanent_uuid: "1fda72425b8941788641e15a3b80c068" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46683 } }
I20260927 21:26:40.686671 1445 leader_election.cc:288] T 52c72a152d1446bda39e4f47511a8840 P 85aa45aa04694932af2607302c8bde36 [CANDIDATE]: Term 1 election: Requested vote from peers a3c4ab3785dc449f83860d5d51b860ae (127.0.97.2:42913), 1fda72425b8941788641e15a3b80c068 (127.0.97.3:46683)
I20260927 21:26:40.687122 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52c72a152d1446bda39e4f47511a8840" candidate_uuid: "85aa45aa04694932af2607302c8bde36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c4ab3785dc449f83860d5d51b860ae"
I20260927 21:26:40.687214 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52c72a152d1446bda39e4f47511a8840" candidate_uuid: "85aa45aa04694932af2607302c8bde36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fda72425b8941788641e15a3b80c068"
I20260927 21:26:40.687310 1242 raft_consensus.cc:3037] T 52c72a152d1446bda39e4f47511a8840 P a3c4ab3785dc449f83860d5d51b860ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:40.687440 1375 raft_consensus.cc:3037] T 52c72a152d1446bda39e4f47511a8840 P 1fda72425b8941788641e15a3b80c068 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:40.689769 1242 raft_consensus.cc:2454] T 52c72a152d1446bda39e4f47511a8840 P a3c4ab3785dc449f83860d5d51b860ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85aa45aa04694932af2607302c8bde36 in term 1.
I20260927 21:26:40.690320 1042 leader_election.cc:302] T 52c72a152d1446bda39e4f47511a8840 P 85aa45aa04694932af2607302c8bde36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85aa45aa04694932af2607302c8bde36, a3c4ab3785dc449f83860d5d51b860ae; no voters:
I20260927 21:26:40.690533 1375 raft_consensus.cc:2454] T 52c72a152d1446bda39e4f47511a8840 P 1fda72425b8941788641e15a3b80c068 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85aa45aa04694932af2607302c8bde36 in term 1.
I20260927 21:26:40.690629 1445 raft_consensus.cc:2781] T 52c72a152d1446bda39e4f47511a8840 P 85aa45aa04694932af2607302c8bde36 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:26:40.691144 1445 raft_consensus.cc:686] T 52c72a152d1446bda39e4f47511a8840 P 85aa45aa04694932af2607302c8bde36 [term 1 LEADER]: Becoming Leader. State: Replica: 85aa45aa04694932af2607302c8bde36, State: Running, Role: LEADER
I20260927 21:26:40.691668 1445 consensus_queue.cc:237] T 52c72a152d1446bda39e4f47511a8840 P 85aa45aa04694932af2607302c8bde36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aa45aa04694932af2607302c8bde36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40919 } } peers { permanent_uuid: "a3c4ab3785dc449f83860d5d51b860ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42913 } } peers { permanent_uuid: "1fda72425b8941788641e15a3b80c068" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46683 } }
I20260927 21:26:40.696151 963 catalog_manager.cc:5495] T 52c72a152d1446bda39e4f47511a8840 P 85aa45aa04694932af2607302c8bde36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85aa45aa04694932af2607302c8bde36 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "85aa45aa04694932af2607302c8bde36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85aa45aa04694932af2607302c8bde36" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3c4ab3785dc449f83860d5d51b860ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fda72425b8941788641e15a3b80c068" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46683 } health_report { overall_health: UNKNOWN } } }
Sep 27 21:26:40 dist-test-slave-bxm0 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 21:26:40.730969 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:40.715566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51270 (local address 127.0.97.1:40919)
0927 21:26:40.715768 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:40.715775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:40.716243 (+ 468us) server_negotiation.cc:407] Connection header received
0927 21:26:40.716437 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:40.716442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:40.716518 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:40.716627 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:40.718110 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.719216 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.720460 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.720863 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.721004 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:40.725205 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:40.725221 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:40.725238 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:40.725277 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:40.728145 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:40.728828 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:40.728837 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:40.728839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:40.728998 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:40.729387 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:40.729394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:40.729397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:40.729827 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:40.730124 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:40.730416 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 21:26:40.730651 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 21:26:40 dist-test-slave-bxm0 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544400, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:26:40 dist-test-slave-bxm0 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544400, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 21:26:40.776324 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:40.762063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51274 (local address 127.0.97.1:40919)
0927 21:26:40.762353 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:40.762363 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:40.762715 (+ 352us) server_negotiation.cc:407] Connection header received
0927 21:26:40.762923 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:40.762927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:40.762990 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:40.763102 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:40.764382 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.765094 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.766365 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.766762 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.766890 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:40.771231 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:40.771236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:40.771239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:40.771275 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:40.773747 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:40.774388 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:40.774394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:40.774397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:40.774531 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:40.774905 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:40.774912 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:40.774915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:40.775181 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:40.775489 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:40.775726 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 21:26:40.775980 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 21:26:40 dist-test-slave-bxm0 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:26:40 dist-test-slave-bxm0 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 21:26:40.820284 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:40.806628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51288 (local address 127.0.97.1:40919)
0927 21:26:40.806887 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:40.806895 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:40.807163 (+ 268us) server_negotiation.cc:407] Connection header received
0927 21:26:40.807408 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:40.807416 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:40.807501 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:40.807602 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:40.808997 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.809757 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.810954 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.811364 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.811494 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:40.815394 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:40.815400 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:40.815404 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:40.815445 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:40.817857 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:40.818441 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:40.818447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:40.818450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:40.818578 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:40.818926 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:40.818932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:40.818935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:40.819212 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:40.819465 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:40.819742 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 21:26:40.819950 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
W20260927 21:26:40.822026 1068 tablet_service.cc:2908] Rejecting scan request for tablet 52c72a152d1446bda39e4f47511a8840: Uninitialized: clean time has not yet been initialized
I20260927 21:26:40.824515 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 1025
I20260927 21:26:40.837224 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:40.825375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51298 (local address 127.0.97.1:40919)
0927 21:26:40.825536 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:40.825541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:40.825919 (+ 378us) server_negotiation.cc:407] Connection header received
0927 21:26:40.826050 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:40.826053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:40.826108 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:40.826184 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:40.827278 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.828116 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.829226 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.829607 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.829741 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:40.832021 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:40.832025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:40.832028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:40.832058 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:40.834975 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:40.835642 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:40.835646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:40.835648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:40.835764 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:40.836088 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:40.836090 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:40.836092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:40.836317 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:40.836485 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:40.836773 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 21:26:40.836943 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 21:26:40.837886 1148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:26:40.928308 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 1025
I20260927 21:26:40.945750 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 1159
Sep 27 21:26:40 dist-test-slave-bxm0 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 21:26:40.960031 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:40.946505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34162 (local address 127.0.97.2:42913)
0927 21:26:40.946672 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:40.946676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:40.946720 (+ 44us) server_negotiation.cc:407] Connection header received
0927 21:26:40.946859 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:40.946863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:40.946924 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:40.947015 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:40.948144 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.948910 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.950295 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:40.950775 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:40.950886 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:40.954692 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:40.954705 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:40.954721 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:40.954754 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:40.957771 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:40.958348 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:40.958355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:40.958357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:40.958472 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:40.958764 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:40.958767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:40.958768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:40.959112 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:40.959405 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:40.959571 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 21:26:40.959773 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 21:26:40.960685 1282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:26:41.047932 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 1159
I20260927 21:26:41.064786 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 1292
Sep 27 21:26:41 dist-test-slave-bxm0 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 21:26:41.079793 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:41.065651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55378 (local address 127.0.97.3:46683)
0927 21:26:41.065818 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:41.065825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:41.065890 (+ 65us) server_negotiation.cc:407] Connection header received
0927 21:26:41.066091 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:41.066097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:41.066169 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:41.066275 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:41.067489 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:41.068215 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:41.069491 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:41.070024 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:41.070161 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:41.074178 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:41.074194 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:41.074211 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:41.074251 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:41.076909 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:41.077658 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:41.077671 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:41.077677 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:41.077833 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:41.078257 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:41.078262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:41.078266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:41.078621 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:41.078924 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:41.079206 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 21:26:41.079437 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 21:26:41.080525 1415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:26:41.166891 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 1292
I20260927 21:26:41.183161 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 932
I20260927 21:26:41.184327 994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:26:41.291210 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 932
2026-09-27T21:26:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4497 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:26:41 dist-test-slave-bxm0 krb5kdc[1468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:26:41 dist-test-slave-bxm0 krb5kdc[1468](info): set up 2 sockets
Sep 27 21:26:41 dist-test-slave-bxm0 krb5kdc[1468](info): commencing operation
krb5kdc: starting...
W20260927 21:26:42.417789 388 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:26:42 dist-test-slave-bxm0 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:26:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:26:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:26:42.603391 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40507
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:26:42.921525 1481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:26:42.921850 1481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:26:42.921947 1481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:26:42.930743 1481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:26:42.930855 1481 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 21:26:42.930896 1481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:26:42.930963 1481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:26:42.931020 1481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:26:42.931087 1481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:26:42.941303 1481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40507
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:26:42.942631 1481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:26:42.944151 1481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:26:42.956162 1487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:42.956188 1489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:42.956238 1486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:26:42.956835 1481 server_base.cc:1034] running on GCE node
I20260927 21:26:42.957660 1481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:26:42.959242 1481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:26:42.960475 1481 hybrid_clock.cc:648] HybridClock initialized: now 1790544402960418 us; error 37 us; skew 500 ppm
I20260927 21:26:42.960799 1481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:26:42 dist-test-slave-bxm0 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:26:42.969082 1481 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:26:42.970633 1481 webserver.cc:460] Webserver started at http://127.0.97.62:41263/ using document root <none> and password file <none>
I20260927 21:26:42.971355 1481 fs_manager.cc:359] Metadata directory not provided
I20260927 21:26:42.971465 1481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:26:42.971819 1481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:26:42.974825 1481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "38a2333e800441a6a0983e7b07b47387"
format_stamp: "Formatted at 2026-09-27 21:26:42 on dist-test-slave-bxm0"
server_key: "1fabb97ab1877605dfd50a87eca76c7f"
server_key_iv: "d0fda2f8accb841534edf7c69c4cc701"
server_key_version: "encryptionkey@0"
I20260927 21:26:42.975623 1481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "38a2333e800441a6a0983e7b07b47387"
format_stamp: "Formatted at 2026-09-27 21:26:42 on dist-test-slave-bxm0"
server_key: "1fabb97ab1877605dfd50a87eca76c7f"
server_key_iv: "d0fda2f8accb841534edf7c69c4cc701"
server_key_version: "encryptionkey@0"
I20260927 21:26:42.980963 1481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260927 21:26:42.984608 1496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:42.986048 1481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:26:42.986227 1481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "38a2333e800441a6a0983e7b07b47387"
format_stamp: "Formatted at 2026-09-27 21:26:42 on dist-test-slave-bxm0"
server_key: "1fabb97ab1877605dfd50a87eca76c7f"
server_key_iv: "d0fda2f8accb841534edf7c69c4cc701"
server_key_version: "encryptionkey@0"
I20260927 21:26:42.986435 1481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:26:43.005489 1481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:26:43.009580 1481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:26:43.009936 1481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:26:43.032333 1481 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40507
I20260927 21:26:43.032351 1557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40507 every 8 connection(s)
I20260927 21:26:43.033980 1481 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:26:43.038172 1558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:43.041059 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 1481
I20260927 21:26:43.041311 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:26:43.042016 388 external_mini_cluster.cc:1428] Setting key 358193509bad5c2ff5ff20adc68d4655
I20260927 21:26:43.051286 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38a2333e800441a6a0983e7b07b47387: Bootstrap starting.
I20260927 21:26:43.056087 1558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38a2333e800441a6a0983e7b07b47387: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:43.057667 1558 log.cc:824] T 00000000000000000000000000000000 P 38a2333e800441a6a0983e7b07b47387: Log is configured to *not* fsync() on all Append() calls
Sep 27 21:26:43 dist-test-slave-bxm0 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:26:43.061477 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38a2333e800441a6a0983e7b07b47387: No bootstrap required, opened a new log
I20260927 21:26:43.068133 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:43.044205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57314 (local address 127.0.97.62:40507)
0927 21:26:43.045027 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:43.045040 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:43.045082 (+ 42us) server_negotiation.cc:407] Connection header received
0927 21:26:43.046889 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:43.046916 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:43.047387 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:43.047771 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:43.049745 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:43.051085 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:43.052903 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:43.054207 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:43.054421 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:43.060042 (+ 5621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:43.060051 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:43.060068 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:43.060106 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:43.063844 (+ 3738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:43.064575 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:43.064589 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:43.064599 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:43.064775 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:43.065170 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:43.065176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:43.065179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:43.065799 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:43.066109 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:43.066552 (+ 443us) server_negotiation.cc:299] Negotiation successful
0927 21:26:43.066999 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":388,"threads_started":1}
I20260927 21:26:43.070170 1558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38a2333e800441a6a0983e7b07b47387 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a2333e800441a6a0983e7b07b47387" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40507 } }
I20260927 21:26:43.070571 1558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38a2333e800441a6a0983e7b07b47387 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:43.070690 1558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38a2333e800441a6a0983e7b07b47387 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38a2333e800441a6a0983e7b07b47387, State: Initialized, Role: FOLLOWER
I20260927 21:26:43.071372 1558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38a2333e800441a6a0983e7b07b47387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a2333e800441a6a0983e7b07b47387" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40507 } }
I20260927 21:26:43.071657 1558 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38a2333e800441a6a0983e7b07b47387 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:26:43.071810 1558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38a2333e800441a6a0983e7b07b47387 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:26:43.071977 1558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38a2333e800441a6a0983e7b07b47387 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:43.074383 1558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38a2333e800441a6a0983e7b07b47387 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a2333e800441a6a0983e7b07b47387" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40507 } }
I20260927 21:26:43.074926 1558 leader_election.cc:302] T 00000000000000000000000000000000 P 38a2333e800441a6a0983e7b07b47387 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38a2333e800441a6a0983e7b07b47387; no voters:
I20260927 21:26:43.075465 1558 leader_election.cc:288] T 00000000000000000000000000000000 P 38a2333e800441a6a0983e7b07b47387 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:26:43.075591 1563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38a2333e800441a6a0983e7b07b47387 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:26:43.076287 1563 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38a2333e800441a6a0983e7b07b47387 [term 1 LEADER]: Becoming Leader. State: Replica: 38a2333e800441a6a0983e7b07b47387, State: Running, Role: LEADER
I20260927 21:26:43.076798 1563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38a2333e800441a6a0983e7b07b47387 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a2333e800441a6a0983e7b07b47387" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40507 } }
I20260927 21:26:43.077508 1558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38a2333e800441a6a0983e7b07b47387 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:26:43.080029 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38a2333e800441a6a0983e7b07b47387 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38a2333e800441a6a0983e7b07b47387" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a2333e800441a6a0983e7b07b47387" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40507 } } }
I20260927 21:26:43.080390 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38a2333e800441a6a0983e7b07b47387 [sys.catalog]: This master's current role is: LEADER
I20260927 21:26:43.079756 1565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38a2333e800441a6a0983e7b07b47387 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38a2333e800441a6a0983e7b07b47387. Latest consensus state: current_term: 1 leader_uuid: "38a2333e800441a6a0983e7b07b47387" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a2333e800441a6a0983e7b07b47387" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40507 } } }
I20260927 21:26:43.081190 1565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38a2333e800441a6a0983e7b07b47387 [sys.catalog]: This master's current role is: LEADER
I20260927 21:26:43.086197 1572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:26:43.091940 1572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:26:43.101827 1572 catalog_manager.cc:1347] Generated new cluster ID: e9108bf82a68418ea986e132d9ccdaef
I20260927 21:26:43.101961 1572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:26:43.118212 1572 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:26:43.119539 1572 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:26:43.125451 1572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38a2333e800441a6a0983e7b07b47387: Generated new TSK 0
I20260927 21:26:43.126327 1572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 21:26:43.179436 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40507
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:26:43.514122 1584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:26:43.514416 1584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:26:43.514577 1584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:26:43.521934 1584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:26:43.522060 1584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:26:43.522265 1584 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:26:43.522374 1584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:26:43.532347 1584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:26:43.533793 1584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:26:43.535173 1584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:26:43.548063 1589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:43.548053 1592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:43.548082 1590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:26:43.549163 1584 server_base.cc:1034] running on GCE node
I20260927 21:26:43.549959 1584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:26:43.550927 1584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:26:43.552169 1584 hybrid_clock.cc:648] HybridClock initialized: now 1790544403552090 us; error 64 us; skew 500 ppm
I20260927 21:26:43.552522 1584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:26:43 dist-test-slave-bxm0 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:26:43.555915 1584 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 21:26:43.557418 1584 webserver.cc:460] Webserver started at http://127.0.97.1:43601/ using document root <none> and password file <none>
I20260927 21:26:43.558277 1584 fs_manager.cc:359] Metadata directory not provided
I20260927 21:26:43.558403 1584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:26:43.558734 1584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:26:43.561882 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-0/data/instance:
uuid: "529ae2624c744fd68bb2affef222bf5a"
format_stamp: "Formatted at 2026-09-27 21:26:43 on dist-test-slave-bxm0"
server_key: "425e298550748647f4058d5115b06db5"
server_key_iv: "faf758da6ccf50082ac74771c971a322"
server_key_version: "encryptionkey@0"
I20260927 21:26:43.562747 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-0/wal/instance:
uuid: "529ae2624c744fd68bb2affef222bf5a"
format_stamp: "Formatted at 2026-09-27 21:26:43 on dist-test-slave-bxm0"
server_key: "425e298550748647f4058d5115b06db5"
server_key_iv: "faf758da6ccf50082ac74771c971a322"
server_key_version: "encryptionkey@0"
I20260927 21:26:43.568172 1584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:26:43.571744 1599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:43.573254 1584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:26:43.573441 1584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "529ae2624c744fd68bb2affef222bf5a"
format_stamp: "Formatted at 2026-09-27 21:26:43 on dist-test-slave-bxm0"
server_key: "425e298550748647f4058d5115b06db5"
server_key_iv: "faf758da6ccf50082ac74771c971a322"
server_key_version: "encryptionkey@0"
I20260927 21:26:43.573725 1584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:26:43.589417 1584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:26:43.593678 1584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:26:43.593946 1584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:26:43.596585 1584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:26:43.596688 1584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:43.596814 1584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:26:43.596889 1584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:26:43 dist-test-slave-bxm0 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:26:43.623646 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:43.597294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52737 (local address 127.0.97.62:40507)
0927 21:26:43.597779 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:43.597784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:43.599294 (+ 1510us) server_negotiation.cc:407] Connection header received
0927 21:26:43.601579 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:43.601585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:43.601722 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:43.601851 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:43.604840 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:43.606039 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:43.609307 (+ 3268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:43.609890 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:43.610037 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:43.615775 (+ 5738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:43.615783 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:43.615790 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:43.615843 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:43.619363 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:43.620351 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:43.620358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:43.620361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:43.620487 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:43.620971 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:43.620977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:43.620980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:43.621405 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:43.621568 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:43.622746 (+ 1178us) server_negotiation.cc:299] Negotiation successful
0927 21:26:43.623087 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":259,"threads_started":1}
I20260927 21:26:43.624838 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:43.598010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40507 (local address 127.0.97.1:52737)
0927 21:26:43.599137 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:43.599181 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:43.601275 (+ 2094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:43.602181 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:43.602196 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:43.603270 (+ 1074us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:26:43.604571 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:43.604621 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:43.606234 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:43.606246 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:43.609113 (+ 2867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:43.609137 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:43.610197 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:43.610211 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:43.610591 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:43.611468 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:26:43.611493 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:26:43.615417 (+ 3924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:26:43.619555 (+ 4138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:43.619568 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:43.619595 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:43.620223 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:43.620614 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:43.620619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:43.620623 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:43.620836 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:43.621595 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:26:43.621631 (+ 36us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:26:43.622235 (+ 604us) client_negotiation.cc:769] Sending connection context
0927 21:26:43.622608 (+ 373us) client_negotiation.cc:240] Negotiation successful
0927 21:26:43.623200 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":405,"threads_started":1}
I20260927 21:26:43.654140 1584 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43365
I20260927 21:26:43.654140 1719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43365 every 8 connection(s)
I20260927 21:26:43.655647 1584 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:26:43.657941 1720 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40507
I20260927 21:26:43.658185 1720 heartbeater.cc:464] Registering TS with master...
I20260927 21:26:43.658865 1720 heartbeater.cc:511] Master 127.0.97.62:40507 requested a full tablet report, sending...
I20260927 21:26:43.660097 1702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:43.644522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40507 (local address 127.0.97.1:58321)
0927 21:26:43.645283 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:43.645311 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:43.645507 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:43.645911 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:43.645919 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:43.646665 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:26:43.647143 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:43.647161 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:43.648275 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:43.648283 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:43.650693 (+ 2410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:43.650718 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:43.651951 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:43.651956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:43.652087 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:43.652780 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:26:43.652797 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:26:43.653850 (+ 1053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:26:43.657318 (+ 3468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:43.657323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:43.657328 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:43.657789 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:43.658216 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:43.658222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:43.658228 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:43.658367 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:43.658888 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:26:43.658893 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:26:43.659098 (+ 205us) client_negotiation.cc:769] Sending connection context
0927 21:26:43.659225 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 21:26:43.659590 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":379,"threads_started":1}
I20260927 21:26:43.660068 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 1584
I20260927 21:26:43.660295 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-0/wal/instance
I20260927 21:26:43.660905 388 external_mini_cluster.cc:1428] Setting key 687403af7a5eac6dde2fa77b3f9a479f
I20260927 21:26:43.660918 1513 ts_manager.cc:194] Registered new tserver with Master: 529ae2624c744fd68bb2affef222bf5a (127.0.97.1:43365)
I20260927 21:26:43.661774 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:43.644570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58321 (local address 127.0.97.62:40507)
0927 21:26:43.644815 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:43.644825 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:43.645430 (+ 605us) server_negotiation.cc:407] Connection header received
0927 21:26:43.645689 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:43.645693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:43.645757 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:43.645843 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:43.647343 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:43.648111 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:43.650925 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:43.651559 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:43.651709 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:43.654047 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:43.654059 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:43.654062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:43.654095 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:43.657157 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:43.657918 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:43.657924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:43.657928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:43.658091 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:43.658486 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:43.658491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:43.658494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:43.658768 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:43.658917 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:43.659343 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 21:26:43.659575 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 21:26:43.662660 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52737
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 21:26:43.702000 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40507
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:26:44.028541 1724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:26:44.028909 1724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:26:44.029071 1724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:26:44.040293 1724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:26:44.040452 1724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:26:44.040586 1724 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:26:44.040715 1724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:26:44.056509 1724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:26:44.057901 1724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:26:44.059268 1724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:26:44.072907 1732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:44.072865 1730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:44.072932 1729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:26:44.074280 1724 server_base.cc:1034] running on GCE node
I20260927 21:26:44.074949 1724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:26:44.075961 1724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:26:44.077224 1724 hybrid_clock.cc:648] HybridClock initialized: now 1790544404077151 us; error 52 us; skew 500 ppm
I20260927 21:26:44.077654 1724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:26:44 dist-test-slave-bxm0 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:26:44.080895 1724 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 21:26:44.082434 1724 webserver.cc:460] Webserver started at http://127.0.97.2:38899/ using document root <none> and password file <none>
I20260927 21:26:44.083169 1724 fs_manager.cc:359] Metadata directory not provided
I20260927 21:26:44.083295 1724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:26:44.083624 1724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:26:44.086710 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-1/data/instance:
uuid: "06aca3f926d248c5a042cf7b5e67bc68"
format_stamp: "Formatted at 2026-09-27 21:26:44 on dist-test-slave-bxm0"
server_key: "2110d231dd8b48ad153a4f8bfbe77a71"
server_key_iv: "fb6186419897356ca95c461bada60897"
server_key_version: "encryptionkey@0"
I20260927 21:26:44.087553 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-1/wal/instance:
uuid: "06aca3f926d248c5a042cf7b5e67bc68"
format_stamp: "Formatted at 2026-09-27 21:26:44 on dist-test-slave-bxm0"
server_key: "2110d231dd8b48ad153a4f8bfbe77a71"
server_key_iv: "fb6186419897356ca95c461bada60897"
server_key_version: "encryptionkey@0"
I20260927 21:26:44.093416 1724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 21:26:44.097138 1739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:44.098614 1724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:26:44.098807 1724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "06aca3f926d248c5a042cf7b5e67bc68"
format_stamp: "Formatted at 2026-09-27 21:26:44 on dist-test-slave-bxm0"
server_key: "2110d231dd8b48ad153a4f8bfbe77a71"
server_key_iv: "fb6186419897356ca95c461bada60897"
server_key_version: "encryptionkey@0"
I20260927 21:26:44.099053 1724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:26:44.133108 1724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:26:44.137887 1724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:26:44.138167 1724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:26:44.140995 1724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:26:44.141094 1724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:44.141218 1724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:26:44.141294 1724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:26:44 dist-test-slave-bxm0 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:26:44.168154 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:44.141640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53317 (local address 127.0.97.62:40507)
0927 21:26:44.141869 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:44.141877 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:44.143365 (+ 1488us) server_negotiation.cc:407] Connection header received
0927 21:26:44.145639 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:44.145646 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:44.145748 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:44.145868 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:44.148531 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.149837 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.153322 (+ 3485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.154070 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.154225 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:44.160065 (+ 5840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:44.160071 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:44.160075 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:44.160115 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:44.164300 (+ 4185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:44.165217 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:44.165226 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:44.165232 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:44.165425 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:44.166053 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:44.166061 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:44.166066 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:44.166520 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:44.166678 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:44.167513 (+ 835us) server_negotiation.cc:299] Negotiation successful
0927 21:26:44.167762 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 21:26:44.169348 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:44.142352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40507 (local address 127.0.97.2:53317)
0927 21:26:44.143182 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:44.143221 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:44.145264 (+ 2043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:44.146180 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:44.146199 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:44.147192 (+ 993us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:26:44.148276 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:44.148310 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.150069 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.150083 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:44.153095 (+ 3012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:44.153126 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.154367 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.154380 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:44.154657 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:44.155579 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:26:44.155620 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:26:44.159737 (+ 4117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:26:44.164534 (+ 4797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:44.164543 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:44.164560 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:44.165069 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:44.165558 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:44.165562 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:44.165565 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:44.165895 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:44.166686 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:26:44.166699 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:26:44.167101 (+ 402us) client_negotiation.cc:769] Sending connection context
0927 21:26:44.167396 (+ 295us) client_negotiation.cc:240] Negotiation successful
0927 21:26:44.168019 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":342,"threads_started":1}
I20260927 21:26:44.194232 1724 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35677
I20260927 21:26:44.194290 1858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35677 every 8 connection(s)
I20260927 21:26:44.195523 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:44.176467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43521 (local address 127.0.97.62:40507)
0927 21:26:44.176728 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:44.176737 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:44.177417 (+ 680us) server_negotiation.cc:407] Connection header received
0927 21:26:44.177786 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:44.177795 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:44.177884 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:44.178026 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:44.179667 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.180765 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.183812 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.184435 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.184550 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:44.187953 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:44.187961 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:44.187968 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:44.188037 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:44.192034 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:44.193082 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:44.193088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:44.193091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:44.193231 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:44.193818 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:44.193824 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:44.193827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:44.194332 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:44.194478 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:44.194944 (+ 466us) server_negotiation.cc:299] Negotiation successful
0927 21:26:44.195177 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 21:26:44.195653 1813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:44.176424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40507 (local address 127.0.97.2:43521)
0927 21:26:44.177268 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:44.177286 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:44.177482 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:44.178141 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:44.178155 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:44.179015 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:26:44.179512 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:44.179531 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.180952 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.180962 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:44.183586 (+ 2624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:44.183621 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.184691 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.184699 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:44.184922 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:44.185742 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:26:44.185775 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:26:44.187596 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:26:44.192180 (+ 4584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:44.192186 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:44.192190 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:44.192892 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:44.193419 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:44.193427 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:44.193431 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:44.193667 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:44.194538 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:26:44.194542 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:26:44.194696 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 21:26:44.194840 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 21:26:44.195212 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":446,"threads_started":1}
I20260927 21:26:44.195710 1724 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:26:44.198033 1859 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40507
I20260927 21:26:44.199223 1859 heartbeater.cc:464] Registering TS with master...
I20260927 21:26:44.199927 1859 heartbeater.cc:511] Master 127.0.97.62:40507 requested a full tablet report, sending...
I20260927 21:26:44.201012 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 1724
I20260927 21:26:44.201218 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-1/wal/instance
I20260927 21:26:44.201602 1513 ts_manager.cc:194] Registered new tserver with Master: 06aca3f926d248c5a042cf7b5e67bc68 (127.0.97.2:35677)
I20260927 21:26:44.201818 388 external_mini_cluster.cc:1428] Setting key 0b3af81bf7a162873f1065a1d1cd505b
I20260927 21:26:44.202817 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53317
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 21:26:44.242046 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40507
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:26:44.566933 1863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:26:44.567282 1863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:26:44.567454 1863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:26:44.574985 1863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:26:44.575160 1863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:26:44.575299 1863 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:26:44.575428 1863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:26:44.585553 1863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:26:44.586923 1863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:26:44.588374 1863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:26:44.601884 1868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:44.601799 1869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:44.601830 1871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:26:44.603072 1863 server_base.cc:1034] running on GCE node
I20260927 21:26:44.603796 1863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:26:44.604738 1863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:26:44.605957 1863 hybrid_clock.cc:648] HybridClock initialized: now 1790544404605869 us; error 67 us; skew 500 ppm
I20260927 21:26:44.606261 1863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:26:44 dist-test-slave-bxm0 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:26:44.609359 1863 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 21:26:44.610978 1863 webserver.cc:460] Webserver started at http://127.0.97.3:43829/ using document root <none> and password file <none>
I20260927 21:26:44.611704 1863 fs_manager.cc:359] Metadata directory not provided
I20260927 21:26:44.611830 1863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:26:44.612159 1863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:26:44.615196 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-2/data/instance:
uuid: "8e96fc7b50264245b05737d64a54e118"
format_stamp: "Formatted at 2026-09-27 21:26:44 on dist-test-slave-bxm0"
server_key: "b1b2ae22ba8e81826f1918f6892f2a0a"
server_key_iv: "f30f936d15c84b6f0b620040da6a293c"
server_key_version: "encryptionkey@0"
I20260927 21:26:44.616044 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8e96fc7b50264245b05737d64a54e118"
format_stamp: "Formatted at 2026-09-27 21:26:44 on dist-test-slave-bxm0"
server_key: "b1b2ae22ba8e81826f1918f6892f2a0a"
server_key_iv: "f30f936d15c84b6f0b620040da6a293c"
server_key_version: "encryptionkey@0"
I20260927 21:26:44.621379 1863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 21:26:44.624956 1878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:44.626338 1863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:26:44.626511 1863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "8e96fc7b50264245b05737d64a54e118"
format_stamp: "Formatted at 2026-09-27 21:26:44 on dist-test-slave-bxm0"
server_key: "b1b2ae22ba8e81826f1918f6892f2a0a"
server_key_iv: "f30f936d15c84b6f0b620040da6a293c"
server_key_version: "encryptionkey@0"
I20260927 21:26:44.626740 1863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:26:44.642130 1863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:26:44.646260 1863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:26:44.646539 1863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:26:44.649252 1863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:26:44.649375 1863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:44.649513 1863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:26:44.649591 1863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:44.665822 1720 heartbeater.cc:503] Master 127.0.97.62:40507 was elected leader, sending a full tablet report...
Sep 27 21:26:44 dist-test-slave-bxm0 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:26:44.676234 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:44.650194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55127 (local address 127.0.97.62:40507)
0927 21:26:44.650380 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:44.650387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:44.651940 (+ 1553us) server_negotiation.cc:407] Connection header received
0927 21:26:44.654629 (+ 2689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:44.654635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:44.654744 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:44.654824 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:44.657385 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.658511 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.661131 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.661798 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.661957 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:44.668359 (+ 6402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:44.668366 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:44.668371 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:44.668411 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:44.672247 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:44.673140 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:44.673148 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:44.673153 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:44.673329 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:44.674036 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:44.674043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:44.674048 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:44.674405 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:44.674573 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:44.675796 (+ 1223us) server_negotiation.cc:299] Negotiation successful
0927 21:26:44.675985 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 21:26:44.678385 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:44.650793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40507 (local address 127.0.97.3:55127)
0927 21:26:44.651723 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:44.651780 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:44.654256 (+ 2476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:44.655388 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:44.655403 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:44.656224 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:26:44.657139 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:44.657175 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.658757 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.658765 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:44.660952 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:44.660964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.662127 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.662138 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:44.662506 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:44.663530 (+ 1024us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:26:44.663574 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:26:44.668117 (+ 4543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:26:44.672406 (+ 4289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:44.672419 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:44.672439 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:44.672992 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:44.673478 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:44.673483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:44.673487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:44.673848 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:44.674579 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:26:44.674587 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:26:44.675224 (+ 637us) client_negotiation.cc:769] Sending connection context
0927 21:26:44.675648 (+ 424us) client_negotiation.cc:240] Negotiation successful
0927 21:26:44.676416 (+ 768us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":414,"threads_started":1}
I20260927 21:26:44.703781 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:44.688762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43565 (local address 127.0.97.62:40507)
0927 21:26:44.688952 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:44.688961 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:44.689584 (+ 623us) server_negotiation.cc:407] Connection header received
0927 21:26:44.689869 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:44.689875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:44.689965 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:44.690067 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:44.691684 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.692925 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.694816 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.695262 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.695407 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:44.697207 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:44.697213 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:44.697217 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:44.697256 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:44.700613 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:44.701564 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:44.701571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:44.701577 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:44.701761 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:44.702250 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:44.702255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:44.702258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:44.702615 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:44.702799 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:44.703369 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 21:26:44.703550 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 21:26:44.704394 1969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:44.688834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40507 (local address 127.0.97.3:43565)
0927 21:26:44.689472 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:44.689493 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:44.689692 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:44.690257 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:44.690264 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:44.691192 (+ 928us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:26:44.691535 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:44.691545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.693108 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.693116 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:44.694658 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:44.694670 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.695451 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.695457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:44.695629 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:44.696282 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:26:44.696299 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:26:44.697014 (+ 715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:26:44.700739 (+ 3725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:44.700744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:44.700748 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:44.701362 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:44.701937 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:44.701943 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:44.701946 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:44.702129 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:44.702782 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:26:44.702788 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:26:44.703057 (+ 269us) client_negotiation.cc:769] Sending connection context
0927 21:26:44.703275 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 21:26:44.703736 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":343,"threads_started":1}
I20260927 21:26:44.709182 1863 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42209
I20260927 21:26:44.709227 1997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42209 every 8 connection(s)
I20260927 21:26:44.710527 1863 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:26:44.712563 1998 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40507
I20260927 21:26:44.712785 1998 heartbeater.cc:464] Registering TS with master...
I20260927 21:26:44.713407 1998 heartbeater.cc:511] Master 127.0.97.62:40507 requested a full tablet report, sending...
I20260927 21:26:44.714744 1512 ts_manager.cc:194] Registered new tserver with Master: 8e96fc7b50264245b05737d64a54e118 (127.0.97.3:42209)
I20260927 21:26:44.715596 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55127
I20260927 21:26:44.720125 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 1863
I20260927 21:26:44.720273 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-2/wal/instance
I20260927 21:26:44.720634 388 external_mini_cluster.cc:1428] Setting key 9b98840890a4aba8453332dca3050020
I20260927 21:26:44.723335 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 21:26:44 dist-test-slave-bxm0 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544404, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:26:44 dist-test-slave-bxm0 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544404, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:26:44.768302 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:44.754380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57318 (local address 127.0.97.62:40507)
0927 21:26:44.754570 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:44.754578 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:44.754940 (+ 362us) server_negotiation.cc:407] Connection header received
0927 21:26:44.755148 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:44.755152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:44.755216 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:44.755328 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:44.756659 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.757398 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.758704 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.759103 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.759234 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:44.763072 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:44.763078 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:44.763081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:44.763120 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:44.765678 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:44.766334 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:44.766340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:44.766344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:44.766474 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:44.766852 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:44.766858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:44.766862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:44.767133 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:44.767375 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:44.767689 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 21:26:44.767910 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 21:26:44.772534 1512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:26:44.779774 1512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:26:44.809121 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:44.791941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43365 (local address 127.0.0.1:42752)
0927 21:26:44.798964 (+ 7023us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:44.799013 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:44.799302 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:44.800452 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:44.800460 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:44.800498 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:26:44.801045 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:44.801057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.803056 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.803062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:44.805696 (+ 2634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:44.805710 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.807980 (+ 2270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.807987 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:44.808264 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:44.808321 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 21:26:44.808466 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 21:26:44.808682 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6452,"mutex_wait_us":28,"thread_start_us":6511,"threads_started":1}
I20260927 21:26:44.809872 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:44.792623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41064 (local address 127.0.97.3:42209)
0927 21:26:44.793580 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:44.793594 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:44.799422 (+ 5828us) server_negotiation.cc:407] Connection header received
0927 21:26:44.799810 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:44.799818 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:44.800013 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:44.800198 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:26:44.801030 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.802826 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.806552 (+ 3726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.808061 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.808164 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:44.808398 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:44.809354 (+ 956us) server_negotiation.cc:299] Negotiation successful
0927 21:26:44.809532 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":559,"threads_started":1}
I20260927 21:26:44.810043 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:44.792115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42752 (local address 127.0.97.1:43365)
0927 21:26:44.798214 (+ 6099us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:44.798228 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:44.799124 (+ 896us) server_negotiation.cc:407] Connection header received
0927 21:26:44.799551 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:44.799562 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:44.799900 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:44.800118 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:26:44.801225 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.802892 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.805876 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.807461 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.807605 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:44.807816 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:44.809537 (+ 1721us) server_negotiation.cc:299] Negotiation successful
0927 21:26:44.809759 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5861,"thread_start_us":5705,"threads_started":1}
I20260927 21:26:44.810621 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:44.792017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42209 (local address 127.0.0.1:41064)
0927 21:26:44.799306 (+ 7289us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:44.799324 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:44.799525 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:44.800207 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:44.800214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:44.800244 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:26:44.800754 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:44.800768 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.804184 (+ 3416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.804189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:44.806363 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:44.806380 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.808906 (+ 2526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.808913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:44.809127 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:44.809172 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 21:26:44.810141 (+ 969us) client_negotiation.cc:240] Negotiation successful
0927 21:26:44.810300 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7176,"thread_start_us":6227,"threads_started":1}
I20260927 21:26:44.814232 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:44.799398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60942 (local address 127.0.97.2:35677)
0927 21:26:44.806502 (+ 7104us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:44.806512 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:44.806945 (+ 433us) server_negotiation.cc:407] Connection header received
0927 21:26:44.807168 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:44.807174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:44.807383 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:44.807597 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:26:44.808467 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.809784 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.811829 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.813190 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.813327 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:44.813508 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:44.813790 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 21:26:44.813948 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6939,"thread_start_us":6806,"threads_started":1}
I20260927 21:26:44.813786 1650 tablet_service.cc:1467] Processing CreateTablet for tablet 48d70dce98a14283a2d2ef89e92ae18d (DEFAULT_TABLE table=test-table [id=ebf58fadba394fc89393bdd66a964015]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:26:44.814188 1928 tablet_service.cc:1467] Processing CreateTablet for tablet 48d70dce98a14283a2d2ef89e92ae18d (DEFAULT_TABLE table=test-table [id=ebf58fadba394fc89393bdd66a964015]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:26:44.815359 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:44.800721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35677 (local address 127.0.0.1:60942)
0927 21:26:44.806809 (+ 6088us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:44.806831 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:44.806986 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:44.807759 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:44.807762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:44.807780 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:26:44.808258 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:44.808272 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.809916 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.809922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:44.811683 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:44.811696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.813402 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.813408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:44.813579 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:44.813636 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 21:26:44.813722 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 21:26:44.813867 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5969,"mutex_wait_us":51,"thread_start_us":5829,"threads_started":1}
I20260927 21:26:44.815604 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48d70dce98a14283a2d2ef89e92ae18d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:44.815497 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48d70dce98a14283a2d2ef89e92ae18d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:44.818769 1789 tablet_service.cc:1467] Processing CreateTablet for tablet 48d70dce98a14283a2d2ef89e92ae18d (DEFAULT_TABLE table=test-table [id=ebf58fadba394fc89393bdd66a964015]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:26:44.820264 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48d70dce98a14283a2d2ef89e92ae18d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:44.827123 2017 tablet_bootstrap.cc:492] T 48d70dce98a14283a2d2ef89e92ae18d P 529ae2624c744fd68bb2affef222bf5a: Bootstrap starting.
I20260927 21:26:44.830379 2018 tablet_bootstrap.cc:492] T 48d70dce98a14283a2d2ef89e92ae18d P 8e96fc7b50264245b05737d64a54e118: Bootstrap starting.
I20260927 21:26:44.831799 2017 tablet_bootstrap.cc:654] T 48d70dce98a14283a2d2ef89e92ae18d P 529ae2624c744fd68bb2affef222bf5a: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:44.833642 2017 log.cc:824] T 48d70dce98a14283a2d2ef89e92ae18d P 529ae2624c744fd68bb2affef222bf5a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:26:44.834523 2018 tablet_bootstrap.cc:654] T 48d70dce98a14283a2d2ef89e92ae18d P 8e96fc7b50264245b05737d64a54e118: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:44.835784 2018 log.cc:824] T 48d70dce98a14283a2d2ef89e92ae18d P 8e96fc7b50264245b05737d64a54e118: Log is configured to *not* fsync() on all Append() calls
I20260927 21:26:44.836980 2017 tablet_bootstrap.cc:492] T 48d70dce98a14283a2d2ef89e92ae18d P 529ae2624c744fd68bb2affef222bf5a: No bootstrap required, opened a new log
I20260927 21:26:44.837282 2017 ts_tablet_manager.cc:1397] T 48d70dce98a14283a2d2ef89e92ae18d P 529ae2624c744fd68bb2affef222bf5a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260927 21:26:44.837745 2020 tablet_bootstrap.cc:492] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68: Bootstrap starting.
I20260927 21:26:44.838743 2018 tablet_bootstrap.cc:492] T 48d70dce98a14283a2d2ef89e92ae18d P 8e96fc7b50264245b05737d64a54e118: No bootstrap required, opened a new log
I20260927 21:26:44.839015 2018 ts_tablet_manager.cc:1397] T 48d70dce98a14283a2d2ef89e92ae18d P 8e96fc7b50264245b05737d64a54e118: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260927 21:26:44.840435 2020 tablet_bootstrap.cc:654] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:44.841760 2020 log.cc:824] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68: Log is configured to *not* fsync() on all Append() calls
I20260927 21:26:44.844576 2020 tablet_bootstrap.cc:492] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68: No bootstrap required, opened a new log
I20260927 21:26:44.844831 2020 ts_tablet_manager.cc:1397] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20260927 21:26:44.848331 2017 raft_consensus.cc:348] T 48d70dce98a14283a2d2ef89e92ae18d P 529ae2624c744fd68bb2affef222bf5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } }
I20260927 21:26:44.848835 2017 raft_consensus.cc:374] T 48d70dce98a14283a2d2ef89e92ae18d P 529ae2624c744fd68bb2affef222bf5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:44.848930 2017 raft_consensus.cc:729] T 48d70dce98a14283a2d2ef89e92ae18d P 529ae2624c744fd68bb2affef222bf5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529ae2624c744fd68bb2affef222bf5a, State: Initialized, Role: FOLLOWER
I20260927 21:26:44.849588 2017 consensus_queue.cc:260] T 48d70dce98a14283a2d2ef89e92ae18d P 529ae2624c744fd68bb2affef222bf5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } }
I20260927 21:26:44.850163 2018 raft_consensus.cc:348] T 48d70dce98a14283a2d2ef89e92ae18d P 8e96fc7b50264245b05737d64a54e118 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } }
I20260927 21:26:44.851403 2017 ts_tablet_manager.cc:1428] T 48d70dce98a14283a2d2ef89e92ae18d P 529ae2624c744fd68bb2affef222bf5a: Time spent starting tablet: real 0.014s user 0.007s sys 0.008s
I20260927 21:26:44.851737 2018 raft_consensus.cc:374] T 48d70dce98a14283a2d2ef89e92ae18d P 8e96fc7b50264245b05737d64a54e118 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:44.851848 2018 raft_consensus.cc:729] T 48d70dce98a14283a2d2ef89e92ae18d P 8e96fc7b50264245b05737d64a54e118 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e96fc7b50264245b05737d64a54e118, State: Initialized, Role: FOLLOWER
I20260927 21:26:44.852501 2018 consensus_queue.cc:260] T 48d70dce98a14283a2d2ef89e92ae18d P 8e96fc7b50264245b05737d64a54e118 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } }
I20260927 21:26:44.853627 1998 heartbeater.cc:503] Master 127.0.97.62:40507 was elected leader, sending a full tablet report...
I20260927 21:26:44.854307 2018 ts_tablet_manager.cc:1428] T 48d70dce98a14283a2d2ef89e92ae18d P 8e96fc7b50264245b05737d64a54e118: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260927 21:26:44.854202 2020 raft_consensus.cc:348] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } }
I20260927 21:26:44.854619 2020 raft_consensus.cc:374] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:44.854725 2020 raft_consensus.cc:729] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06aca3f926d248c5a042cf7b5e67bc68, State: Initialized, Role: FOLLOWER
I20260927 21:26:44.855406 2020 consensus_queue.cc:260] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } }
I20260927 21:26:44.856639 1859 heartbeater.cc:503] Master 127.0.97.62:40507 was elected leader, sending a full tablet report...
I20260927 21:26:44.857273 2020 ts_tablet_manager.cc:1428] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68: Time spent starting tablet: real 0.012s user 0.014s sys 0.000s
W20260927 21:26:44.908288 1721 tablet.cc:2369] T 48d70dce98a14283a2d2ef89e92ae18d P 529ae2624c744fd68bb2affef222bf5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:26:44.947844 1860 tablet.cc:2369] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:26:44.953842 2025 raft_consensus.cc:482] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:26:44.954185 2025 raft_consensus.cc:504] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } }
I20260927 21:26:44.956004 2025 leader_election.cc:288] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 529ae2624c744fd68bb2affef222bf5a (127.0.97.1:43365), 8e96fc7b50264245b05737d64a54e118 (127.0.97.3:42209)
W20260927 21:26:44.962095 1999 tablet.cc:2369] T 48d70dce98a14283a2d2ef89e92ae18d P 8e96fc7b50264245b05737d64a54e118: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:26:44.962944 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:44.956445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37919 (local address 127.0.97.1:43365)
0927 21:26:44.956737 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:44.956746 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:44.957433 (+ 687us) server_negotiation.cc:407] Connection header received
0927 21:26:44.957646 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:44.957651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:44.957746 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:44.957833 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:26:44.958472 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.959262 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.961118 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.961974 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.962138 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:44.962243 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:44.962567 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 21:26:44.962695 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260927 21:26:44.963091 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:44.956222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43365 (local address 127.0.97.2:37919)
0927 21:26:44.957290 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:44.957315 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:44.957478 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:44.957857 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:44.957863 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:44.957886 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:26:44.958319 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:44.958333 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.959391 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.959395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:44.960963 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:44.960975 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.962198 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.962204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:44.962380 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:44.962420 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 21:26:44.962540 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 21:26:44.962678 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"mutex_wait_us":1,"thread_start_us":667,"threads_started":1}
I20260927 21:26:44.963275 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:44.956679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51975 (local address 127.0.97.3:42209)
0927 21:26:44.956957 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:44.956963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:44.957653 (+ 690us) server_negotiation.cc:407] Connection header received
0927 21:26:44.957781 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:44.957785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:44.957881 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:44.958037 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:26:44.958618 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.959408 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.961199 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.962232 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.962352 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:44.962460 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:44.962876 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 21:26:44.963017 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260927 21:26:44.963766 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:44.956541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42209 (local address 127.0.97.2:51975)
0927 21:26:44.957471 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:44.957498 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:44.957667 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:44.958046 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:44.958049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:44.958068 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:26:44.958439 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:44.958452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.959546 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.959549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:44.961016 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:44.961050 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.962527 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.962534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:44.962685 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:44.962717 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 21:26:44.963418 (+ 701us) client_negotiation.cc:240] Negotiation successful
0927 21:26:44.963534 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":21,"thread_start_us":575,"threads_started":1}
I20260927 21:26:44.964658 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48d70dce98a14283a2d2ef89e92ae18d" candidate_uuid: "06aca3f926d248c5a042cf7b5e67bc68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529ae2624c744fd68bb2affef222bf5a" is_pre_election: true
I20260927 21:26:44.965229 1670 raft_consensus.cc:2454] T 48d70dce98a14283a2d2ef89e92ae18d P 529ae2624c744fd68bb2affef222bf5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06aca3f926d248c5a042cf7b5e67bc68 in term 0.
I20260927 21:26:44.965288 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48d70dce98a14283a2d2ef89e92ae18d" candidate_uuid: "06aca3f926d248c5a042cf7b5e67bc68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e96fc7b50264245b05737d64a54e118" is_pre_election: true
I20260927 21:26:44.965840 1948 raft_consensus.cc:2454] T 48d70dce98a14283a2d2ef89e92ae18d P 8e96fc7b50264245b05737d64a54e118 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06aca3f926d248c5a042cf7b5e67bc68 in term 0.
I20260927 21:26:44.965914 1740 leader_election.cc:302] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06aca3f926d248c5a042cf7b5e67bc68, 529ae2624c744fd68bb2affef222bf5a; no voters:
I20260927 21:26:44.966351 2025 raft_consensus.cc:2781] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:26:44.966544 2025 raft_consensus.cc:482] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:26:44.966656 2025 raft_consensus.cc:3037] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:44.968801 2025 raft_consensus.cc:504] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } }
I20260927 21:26:44.969370 2025 leader_election.cc:288] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68 [CANDIDATE]: Term 1 election: Requested vote from peers 529ae2624c744fd68bb2affef222bf5a (127.0.97.1:43365), 8e96fc7b50264245b05737d64a54e118 (127.0.97.3:42209)
I20260927 21:26:44.969875 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48d70dce98a14283a2d2ef89e92ae18d" candidate_uuid: "06aca3f926d248c5a042cf7b5e67bc68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e96fc7b50264245b05737d64a54e118"
I20260927 21:26:44.969882 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48d70dce98a14283a2d2ef89e92ae18d" candidate_uuid: "06aca3f926d248c5a042cf7b5e67bc68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529ae2624c744fd68bb2affef222bf5a"
I20260927 21:26:44.970068 1948 raft_consensus.cc:3037] T 48d70dce98a14283a2d2ef89e92ae18d P 8e96fc7b50264245b05737d64a54e118 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:44.970055 1670 raft_consensus.cc:3037] T 48d70dce98a14283a2d2ef89e92ae18d P 529ae2624c744fd68bb2affef222bf5a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:44.972368 1948 raft_consensus.cc:2454] T 48d70dce98a14283a2d2ef89e92ae18d P 8e96fc7b50264245b05737d64a54e118 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06aca3f926d248c5a042cf7b5e67bc68 in term 1.
I20260927 21:26:44.972458 1670 raft_consensus.cc:2454] T 48d70dce98a14283a2d2ef89e92ae18d P 529ae2624c744fd68bb2affef222bf5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06aca3f926d248c5a042cf7b5e67bc68 in term 1.
I20260927 21:26:44.972836 1741 leader_election.cc:302] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06aca3f926d248c5a042cf7b5e67bc68, 8e96fc7b50264245b05737d64a54e118; no voters:
I20260927 21:26:44.973176 2025 raft_consensus.cc:2781] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:26:44.973678 2025 raft_consensus.cc:686] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68 [term 1 LEADER]: Becoming Leader. State: Replica: 06aca3f926d248c5a042cf7b5e67bc68, State: Running, Role: LEADER
I20260927 21:26:44.974221 2025 consensus_queue.cc:237] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } }
I20260927 21:26:44.979072 1512 catalog_manager.cc:5495] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06aca3f926d248c5a042cf7b5e67bc68 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "06aca3f926d248c5a042cf7b5e67bc68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } health_report { overall_health: HEALTHY } } }
I20260927 21:26:45.001784 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:44.996295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60958 (local address 127.0.97.2:35677)
0927 21:26:44.996509 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:44.996516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:44.996691 (+ 175us) server_negotiation.cc:407] Connection header received
0927 21:26:44.996951 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:44.996955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:44.997030 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:44.997165 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:26:44.997462 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.998224 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:44.999545 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:44.999981 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.000086 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:45.000629 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:26:45.001211 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:26:45.001320 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:45.001430 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 21:26:45.001502 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 21:26:45.010201 1948 raft_consensus.cc:1261] T 48d70dce98a14283a2d2ef89e92ae18d P 8e96fc7b50264245b05737d64a54e118 [term 1 FOLLOWER]: Refusing update from remote peer 06aca3f926d248c5a042cf7b5e67bc68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:26:45.010202 1670 raft_consensus.cc:1261] T 48d70dce98a14283a2d2ef89e92ae18d P 529ae2624c744fd68bb2affef222bf5a [term 1 FOLLOWER]: Refusing update from remote peer 06aca3f926d248c5a042cf7b5e67bc68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:26:45.011198 2025 consensus_queue.cc:1035] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:26:45.011509 2028 consensus_queue.cc:1035] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:26:45.047956 1512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:26:45.048306 1512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:26:45.052238 1512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a2333e800441a6a0983e7b07b47387: Sending DeleteTablet for 3 replicas of tablet 48d70dce98a14283a2d2ef89e92ae18d
I20260927 21:26:45.053485 1650 tablet_service.cc:1514] Processing DeleteTablet for tablet 48d70dce98a14283a2d2ef89e92ae18d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:26:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42752
I20260927 21:26:45.053567 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet 48d70dce98a14283a2d2ef89e92ae18d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:26:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41064
I20260927 21:26:45.054462 2041 tablet_replica.cc:330] T 48d70dce98a14283a2d2ef89e92ae18d P 529ae2624c744fd68bb2affef222bf5a: stopping tablet replica
I20260927 21:26:45.054992 2041 raft_consensus.cc:2229] T 48d70dce98a14283a2d2ef89e92ae18d P 529ae2624c744fd68bb2affef222bf5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:26:45.054981 1789 tablet_service.cc:1514] Processing DeleteTablet for tablet 48d70dce98a14283a2d2ef89e92ae18d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:26:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60942
I20260927 21:26:45.055768 2041 raft_consensus.cc:2258] T 48d70dce98a14283a2d2ef89e92ae18d P 529ae2624c744fd68bb2affef222bf5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:26:45.057356 1512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 21:26:45.058279 2041 ts_tablet_manager.cc:1905] T 48d70dce98a14283a2d2ef89e92ae18d P 529ae2624c744fd68bb2affef222bf5a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 21:26:45.058625 1512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:26:45.065951 2041 ts_tablet_manager.cc:1918] T 48d70dce98a14283a2d2ef89e92ae18d P 529ae2624c744fd68bb2affef222bf5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:26:45.066744 2043 tablet_replica.cc:330] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68: stopping tablet replica
I20260927 21:26:45.067340 2043 raft_consensus.cc:2229] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:26:45.068748 1650 tablet_service.cc:1467] Processing CreateTablet for tablet 93fe9dcfbb034497969f19785ea72cbe (DEFAULT_TABLE table=test-table [id=970d6e2052d24131886c7ca3fec7958b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:26:45.068954 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 93fe9dcfbb034497969f19785ea72cbe (DEFAULT_TABLE table=test-table [id=970d6e2052d24131886c7ca3fec7958b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:26:45.069226 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93fe9dcfbb034497969f19785ea72cbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:45.069106 1788 tablet_service.cc:1467] Processing CreateTablet for tablet 93fe9dcfbb034497969f19785ea72cbe (DEFAULT_TABLE table=test-table [id=970d6e2052d24131886c7ca3fec7958b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:26:45.069681 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93fe9dcfbb034497969f19785ea72cbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:45.069784 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93fe9dcfbb034497969f19785ea72cbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:45.069062 2043 raft_consensus.cc:2258] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:26:45.076452 2017 tablet_bootstrap.cc:492] T 93fe9dcfbb034497969f19785ea72cbe P 529ae2624c744fd68bb2affef222bf5a: Bootstrap starting.
I20260927 21:26:45.078279 2017 tablet_bootstrap.cc:654] T 93fe9dcfbb034497969f19785ea72cbe P 529ae2624c744fd68bb2affef222bf5a: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:45.078953 2042 tablet_replica.cc:330] T 48d70dce98a14283a2d2ef89e92ae18d P 8e96fc7b50264245b05737d64a54e118: stopping tablet replica
I20260927 21:26:45.079430 2042 raft_consensus.cc:2229] T 48d70dce98a14283a2d2ef89e92ae18d P 8e96fc7b50264245b05737d64a54e118 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:26:45.079975 2042 raft_consensus.cc:2258] T 48d70dce98a14283a2d2ef89e92ae18d P 8e96fc7b50264245b05737d64a54e118 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:26:45.080034 2041 log.cc:1196] T 48d70dce98a14283a2d2ef89e92ae18d P 529ae2624c744fd68bb2affef222bf5a: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-0/wal/wals/48d70dce98a14283a2d2ef89e92ae18d
I20260927 21:26:45.080641 2018 tablet_bootstrap.cc:492] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118: Bootstrap starting.
I20260927 21:26:45.082005 2020 tablet_bootstrap.cc:492] T 93fe9dcfbb034497969f19785ea72cbe P 06aca3f926d248c5a042cf7b5e67bc68: Bootstrap starting.
I20260927 21:26:45.082377 2018 tablet_bootstrap.cc:654] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:45.082594 2042 ts_tablet_manager.cc:1905] T 48d70dce98a14283a2d2ef89e92ae18d P 8e96fc7b50264245b05737d64a54e118: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:26:45.083693 2020 tablet_bootstrap.cc:654] T 93fe9dcfbb034497969f19785ea72cbe P 06aca3f926d248c5a042cf7b5e67bc68: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:45.086280 2043 ts_tablet_manager.cc:1905] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:26:45.088276 2017 tablet_bootstrap.cc:492] T 93fe9dcfbb034497969f19785ea72cbe P 529ae2624c744fd68bb2affef222bf5a: No bootstrap required, opened a new log
I20260927 21:26:45.088449 2017 ts_tablet_manager.cc:1397] T 93fe9dcfbb034497969f19785ea72cbe P 529ae2624c744fd68bb2affef222bf5a: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20260927 21:26:45.089231 2017 raft_consensus.cc:348] T 93fe9dcfbb034497969f19785ea72cbe P 529ae2624c744fd68bb2affef222bf5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } }
I20260927 21:26:45.089471 2017 raft_consensus.cc:374] T 93fe9dcfbb034497969f19785ea72cbe P 529ae2624c744fd68bb2affef222bf5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:45.089568 2017 raft_consensus.cc:729] T 93fe9dcfbb034497969f19785ea72cbe P 529ae2624c744fd68bb2affef222bf5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529ae2624c744fd68bb2affef222bf5a, State: Initialized, Role: FOLLOWER
I20260927 21:26:45.089946 2017 consensus_queue.cc:260] T 93fe9dcfbb034497969f19785ea72cbe P 529ae2624c744fd68bb2affef222bf5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } }
I20260927 21:26:45.090902 2042 ts_tablet_manager.cc:1918] T 48d70dce98a14283a2d2ef89e92ae18d P 8e96fc7b50264245b05737d64a54e118: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:26:45.091142 2042 log.cc:1196] T 48d70dce98a14283a2d2ef89e92ae18d P 8e96fc7b50264245b05737d64a54e118: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-2/wal/wals/48d70dce98a14283a2d2ef89e92ae18d
I20260927 21:26:45.091163 2020 tablet_bootstrap.cc:492] T 93fe9dcfbb034497969f19785ea72cbe P 06aca3f926d248c5a042cf7b5e67bc68: No bootstrap required, opened a new log
I20260927 21:26:45.091405 2020 ts_tablet_manager.cc:1397] T 93fe9dcfbb034497969f19785ea72cbe P 06aca3f926d248c5a042cf7b5e67bc68: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260927 21:26:45.091711 2042 ts_tablet_manager.cc:1939] T 48d70dce98a14283a2d2ef89e92ae18d P 8e96fc7b50264245b05737d64a54e118: Deleting consensus metadata
I20260927 21:26:45.092317 2020 raft_consensus.cc:348] T 93fe9dcfbb034497969f19785ea72cbe P 06aca3f926d248c5a042cf7b5e67bc68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } }
I20260927 21:26:45.092618 2020 raft_consensus.cc:374] T 93fe9dcfbb034497969f19785ea72cbe P 06aca3f926d248c5a042cf7b5e67bc68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:45.092780 2020 raft_consensus.cc:729] T 93fe9dcfbb034497969f19785ea72cbe P 06aca3f926d248c5a042cf7b5e67bc68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06aca3f926d248c5a042cf7b5e67bc68, State: Initialized, Role: FOLLOWER
I20260927 21:26:45.092912 2017 ts_tablet_manager.cc:1428] T 93fe9dcfbb034497969f19785ea72cbe P 529ae2624c744fd68bb2affef222bf5a: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 21:26:45.093179 2020 consensus_queue.cc:260] T 93fe9dcfbb034497969f19785ea72cbe P 06aca3f926d248c5a042cf7b5e67bc68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } }
I20260927 21:26:45.093981 2041 ts_tablet_manager.cc:1939] T 48d70dce98a14283a2d2ef89e92ae18d P 529ae2624c744fd68bb2affef222bf5a: Deleting consensus metadata
I20260927 21:26:45.093973 1498 catalog_manager.cc:4841] TS 8e96fc7b50264245b05737d64a54e118 (127.0.97.3:42209): tablet 48d70dce98a14283a2d2ef89e92ae18d (table test-table [id=ebf58fadba394fc89393bdd66a964015]) successfully deleted
I20260927 21:26:45.094484 2020 ts_tablet_manager.cc:1428] T 93fe9dcfbb034497969f19785ea72cbe P 06aca3f926d248c5a042cf7b5e67bc68: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 21:26:45.096241 1497 catalog_manager.cc:4841] TS 529ae2624c744fd68bb2affef222bf5a (127.0.97.1:43365): tablet 48d70dce98a14283a2d2ef89e92ae18d (table test-table [id=ebf58fadba394fc89393bdd66a964015]) successfully deleted
I20260927 21:26:45.099293 2018 tablet_bootstrap.cc:492] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118: No bootstrap required, opened a new log
I20260927 21:26:45.099474 2018 ts_tablet_manager.cc:1397] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.005s
I20260927 21:26:45.100134 2018 raft_consensus.cc:348] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } }
I20260927 21:26:45.100327 2018 raft_consensus.cc:374] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:45.100420 2018 raft_consensus.cc:729] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e96fc7b50264245b05737d64a54e118, State: Initialized, Role: FOLLOWER
I20260927 21:26:45.100683 2018 consensus_queue.cc:260] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } }
I20260927 21:26:45.100916 2043 ts_tablet_manager.cc:1918] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:26:45.101127 2043 log.cc:1196] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-1/wal/wals/48d70dce98a14283a2d2ef89e92ae18d
I20260927 21:26:45.101257 2018 ts_tablet_manager.cc:1428] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 21:26:45.101708 2043 ts_tablet_manager.cc:1939] T 48d70dce98a14283a2d2ef89e92ae18d P 06aca3f926d248c5a042cf7b5e67bc68: Deleting consensus metadata
I20260927 21:26:45.103922 1498 catalog_manager.cc:4841] TS 06aca3f926d248c5a042cf7b5e67bc68 (127.0.97.2:35677): tablet 48d70dce98a14283a2d2ef89e92ae18d (table test-table [id=ebf58fadba394fc89393bdd66a964015]) successfully deleted
I20260927 21:26:45.484292 2024 raft_consensus.cc:482] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:26:45.484567 2024 raft_consensus.cc:504] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } }
I20260927 21:26:45.486608 2024 leader_election.cc:288] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 529ae2624c744fd68bb2affef222bf5a (127.0.97.1:43365), 06aca3f926d248c5a042cf7b5e67bc68 (127.0.97.2:35677)
I20260927 21:26:45.500214 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:45.488064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46433 (local address 127.0.97.2:35677)
0927 21:26:45.488261 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:45.488267 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:45.489890 (+ 1623us) server_negotiation.cc:407] Connection header received
0927 21:26:45.490342 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:45.490350 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:45.490456 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:45.490544 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:26:45.491555 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.493075 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.496577 (+ 3502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.498445 (+ 1868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.498611 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:45.498766 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:45.499537 (+ 771us) server_negotiation.cc:299] Negotiation successful
0927 21:26:45.499762 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 21:26:45.501664 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:45.487609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35677 (local address 127.0.97.3:46433)
0927 21:26:45.489733 (+ 2124us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:45.489775 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:45.490104 (+ 329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:45.490809 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:45.490817 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:45.490858 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:26:45.491291 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:45.491312 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.493345 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.493353 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:45.496307 (+ 2954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:45.496330 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.498971 (+ 2641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.498981 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:45.499277 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:45.499331 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 21:26:45.500888 (+ 1557us) client_negotiation.cc:240] Negotiation successful
0927 21:26:45.501114 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1924,"thread_start_us":1152,"threads_started":1}
I20260927 21:26:45.502911 1814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe9dcfbb034497969f19785ea72cbe" candidate_uuid: "8e96fc7b50264245b05737d64a54e118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06aca3f926d248c5a042cf7b5e67bc68" is_pre_election: true
I20260927 21:26:45.503504 1814 raft_consensus.cc:2454] T 93fe9dcfbb034497969f19785ea72cbe P 06aca3f926d248c5a042cf7b5e67bc68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e96fc7b50264245b05737d64a54e118 in term 0.
I20260927 21:26:45.504602 1880 leader_election.cc:302] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06aca3f926d248c5a042cf7b5e67bc68, 8e96fc7b50264245b05737d64a54e118; no voters:
I20260927 21:26:45.505120 2024 raft_consensus.cc:2781] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:26:45.505285 2024 raft_consensus.cc:482] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:26:45.505400 2024 raft_consensus.cc:3037] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:45.508999 2024 raft_consensus.cc:504] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } }
I20260927 21:26:45.510659 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:45.486720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43365 (local address 127.0.97.3:37241)
0927 21:26:45.488248 (+ 1528us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:45.488274 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:45.488561 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:45.502827 (+ 14266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:45.502832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:45.502863 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:26:45.503277 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:45.503294 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.504594 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.504602 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:45.507230 (+ 2628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:45.507255 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.509634 (+ 2379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.509645 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:45.509899 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:45.509955 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 21:26:45.510037 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 21:26:45.510219 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1346,"thread_start_us":1552,"threads_started":1}
I20260927 21:26:45.510560 1814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe9dcfbb034497969f19785ea72cbe" candidate_uuid: "8e96fc7b50264245b05737d64a54e118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06aca3f926d248c5a042cf7b5e67bc68"
I20260927 21:26:45.510825 1814 raft_consensus.cc:3037] T 93fe9dcfbb034497969f19785ea72cbe P 06aca3f926d248c5a042cf7b5e67bc68 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:45.512050 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:45.486790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37241 (local address 127.0.97.1:43365)
0927 21:26:45.502490 (+ 15700us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:45.502498 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:45.502519 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:26:45.502589 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:45.502594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:45.502681 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:45.502807 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:26:45.503506 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.504454 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.507599 (+ 3145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.509242 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.511170 (+ 1928us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:45.511320 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:45.511385 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 21:26:45.511586 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15419,"thread_start_us":15515,"threads_started":1}
I20260927 21:26:45.514143 1814 raft_consensus.cc:2454] T 93fe9dcfbb034497969f19785ea72cbe P 06aca3f926d248c5a042cf7b5e67bc68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e96fc7b50264245b05737d64a54e118 in term 1.
I20260927 21:26:45.515208 1880 leader_election.cc:302] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06aca3f926d248c5a042cf7b5e67bc68, 8e96fc7b50264245b05737d64a54e118; no voters:
I20260927 21:26:45.515758 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe9dcfbb034497969f19785ea72cbe" candidate_uuid: "8e96fc7b50264245b05737d64a54e118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529ae2624c744fd68bb2affef222bf5a" is_pre_election: true
I20260927 21:26:45.515852 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fe9dcfbb034497969f19785ea72cbe" candidate_uuid: "8e96fc7b50264245b05737d64a54e118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529ae2624c744fd68bb2affef222bf5a"
I20260927 21:26:45.516067 1670 raft_consensus.cc:2454] T 93fe9dcfbb034497969f19785ea72cbe P 529ae2624c744fd68bb2affef222bf5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e96fc7b50264245b05737d64a54e118 in term 0.
I20260927 21:26:45.516222 1669 raft_consensus.cc:3037] T 93fe9dcfbb034497969f19785ea72cbe P 529ae2624c744fd68bb2affef222bf5a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:45.517176 2024 leader_election.cc:288] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118 [CANDIDATE]: Term 1 election: Requested vote from peers 529ae2624c744fd68bb2affef222bf5a (127.0.97.1:43365), 06aca3f926d248c5a042cf7b5e67bc68 (127.0.97.2:35677)
I20260927 21:26:45.517397 2024 raft_consensus.cc:2781] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:26:45.517549 2024 raft_consensus.cc:686] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118 [term 1 LEADER]: Becoming Leader. State: Replica: 8e96fc7b50264245b05737d64a54e118, State: Running, Role: LEADER
I20260927 21:26:45.518000 2024 consensus_queue.cc:237] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } }
I20260927 21:26:45.520406 1669 raft_consensus.cc:2454] T 93fe9dcfbb034497969f19785ea72cbe P 529ae2624c744fd68bb2affef222bf5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e96fc7b50264245b05737d64a54e118 in term 1.
I20260927 21:26:45.531729 1513 catalog_manager.cc:5495] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e96fc7b50264245b05737d64a54e118 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "8e96fc7b50264245b05737d64a54e118" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } health_report { overall_health: HEALTHY } } }
Sep 27 21:26:45 dist-test-slave-bxm0 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:26:45.591542 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:45.575977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40507 (local address 127.0.0.1:57322)
0927 21:26:45.577111 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:45.577138 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:45.577317 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:45.577685 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:45.577689 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:45.578431 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:26:45.578635 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:45.578645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.579774 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.579781 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:45.580747 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:45.580764 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.581461 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.581466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:45.581685 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:45.582473 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:26:45.582494 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:26:45.585105 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:26:45.588437 (+ 3332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:45.588455 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:45.588466 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:45.589312 (+ 846us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:45.589756 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:45.589761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:45.589764 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:45.589971 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:45.590597 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:26:45.590603 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:26:45.590742 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 21:26:45.590874 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 21:26:45.591193 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":957,"thread_start_us":801,"threads_started":1}
I20260927 21:26:45.591984 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:45.576027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57322 (local address 127.0.97.62:40507)
0927 21:26:45.577394 (+ 1367us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:45.577400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:45.577423 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:26:45.577474 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:45.577477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:45.577530 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:45.577667 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:45.578793 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.579648 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.580910 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.581307 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.581439 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:45.585335 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:45.585340 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:45.585343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:45.585375 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:45.588278 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:45.589446 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:45.589450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:45.589452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:45.589594 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:45.590104 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:45.590108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:45.590110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:45.590396 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:45.590531 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:45.591026 (+ 495us) server_negotiation.cc:299] Negotiation successful
0927 21:26:45.591397 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":1244,"thread_start_us":1111,"threads_started":1}
I20260927 21:26:45.597718 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57322:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 21:26:45.599185 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:26:45.607043 1650 tablet_service.cc:1467] Processing CreateTablet for tablet 7220f7e16dbb472b8a99b112b67ef940 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5322b3a515314e8fb8dcaf46cd61eefa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:26:45.607524 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7220f7e16dbb472b8a99b112b67ef940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:45.613081 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 7220f7e16dbb472b8a99b112b67ef940 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5322b3a515314e8fb8dcaf46cd61eefa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:26:45.613075 1788 tablet_service.cc:1467] Processing CreateTablet for tablet 7220f7e16dbb472b8a99b112b67ef940 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5322b3a515314e8fb8dcaf46cd61eefa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:26:45.613577 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7220f7e16dbb472b8a99b112b67ef940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:45.613570 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7220f7e16dbb472b8a99b112b67ef940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:45.622388 2060 tablet_bootstrap.cc:492] T 7220f7e16dbb472b8a99b112b67ef940 P 8e96fc7b50264245b05737d64a54e118: Bootstrap starting.
I20260927 21:26:45.622313 2059 tablet_bootstrap.cc:492] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68: Bootstrap starting.
I20260927 21:26:45.622648 2061 tablet_bootstrap.cc:492] T 7220f7e16dbb472b8a99b112b67ef940 P 529ae2624c744fd68bb2affef222bf5a: Bootstrap starting.
I20260927 21:26:45.625536 2061 tablet_bootstrap.cc:654] T 7220f7e16dbb472b8a99b112b67ef940 P 529ae2624c744fd68bb2affef222bf5a: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:45.631016 2060 tablet_bootstrap.cc:654] T 7220f7e16dbb472b8a99b112b67ef940 P 8e96fc7b50264245b05737d64a54e118: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:45.631112 2059 tablet_bootstrap.cc:654] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:45.631888 2061 tablet_bootstrap.cc:492] T 7220f7e16dbb472b8a99b112b67ef940 P 529ae2624c744fd68bb2affef222bf5a: No bootstrap required, opened a new log
I20260927 21:26:45.632074 2061 ts_tablet_manager.cc:1397] T 7220f7e16dbb472b8a99b112b67ef940 P 529ae2624c744fd68bb2affef222bf5a: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260927 21:26:45.633177 2061 raft_consensus.cc:348] T 7220f7e16dbb472b8a99b112b67ef940 P 529ae2624c744fd68bb2affef222bf5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } }
I20260927 21:26:45.633534 2061 raft_consensus.cc:374] T 7220f7e16dbb472b8a99b112b67ef940 P 529ae2624c744fd68bb2affef222bf5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:45.633672 2061 raft_consensus.cc:729] T 7220f7e16dbb472b8a99b112b67ef940 P 529ae2624c744fd68bb2affef222bf5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529ae2624c744fd68bb2affef222bf5a, State: Initialized, Role: FOLLOWER
I20260927 21:26:45.634054 2061 consensus_queue.cc:260] T 7220f7e16dbb472b8a99b112b67ef940 P 529ae2624c744fd68bb2affef222bf5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } }
I20260927 21:26:45.643180 2059 tablet_bootstrap.cc:492] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68: No bootstrap required, opened a new log
I20260927 21:26:45.643204 2060 tablet_bootstrap.cc:492] T 7220f7e16dbb472b8a99b112b67ef940 P 8e96fc7b50264245b05737d64a54e118: No bootstrap required, opened a new log
I20260927 21:26:45.643412 2059 ts_tablet_manager.cc:1397] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.010s
I20260927 21:26:45.643571 2060 ts_tablet_manager.cc:1397] T 7220f7e16dbb472b8a99b112b67ef940 P 8e96fc7b50264245b05737d64a54e118: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.006s
I20260927 21:26:45.644528 2059 raft_consensus.cc:348] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } }
I20260927 21:26:45.644886 2059 raft_consensus.cc:374] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:45.644598 2060 raft_consensus.cc:348] T 7220f7e16dbb472b8a99b112b67ef940 P 8e96fc7b50264245b05737d64a54e118 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } }
I20260927 21:26:45.645010 2059 raft_consensus.cc:729] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06aca3f926d248c5a042cf7b5e67bc68, State: Initialized, Role: FOLLOWER
I20260927 21:26:45.645138 2060 raft_consensus.cc:374] T 7220f7e16dbb472b8a99b112b67ef940 P 8e96fc7b50264245b05737d64a54e118 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:45.645298 2060 raft_consensus.cc:729] T 7220f7e16dbb472b8a99b112b67ef940 P 8e96fc7b50264245b05737d64a54e118 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e96fc7b50264245b05737d64a54e118, State: Initialized, Role: FOLLOWER
I20260927 21:26:45.645427 2059 consensus_queue.cc:260] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } }
I20260927 21:26:45.645675 2060 consensus_queue.cc:260] T 7220f7e16dbb472b8a99b112b67ef940 P 8e96fc7b50264245b05737d64a54e118 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } }
I20260927 21:26:45.646416 2024 tablet_replica.cc:439] T 7220f7e16dbb472b8a99b112b67ef940 P 8e96fc7b50264245b05737d64a54e118: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } } }
I20260927 21:26:45.646770 2024 tablet_replica.cc:442] T 7220f7e16dbb472b8a99b112b67ef940 P 8e96fc7b50264245b05737d64a54e118: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:26:45.651703 2060 ts_tablet_manager.cc:1428] T 7220f7e16dbb472b8a99b112b67ef940 P 8e96fc7b50264245b05737d64a54e118: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260927 21:26:45.652729 2059 ts_tablet_manager.cc:1428] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68: Time spent starting tablet: real 0.009s user 0.008s sys 0.002s
I20260927 21:26:45.654008 2065 tablet_replica.cc:439] T 7220f7e16dbb472b8a99b112b67ef940 P 529ae2624c744fd68bb2affef222bf5a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } } }
I20260927 21:26:45.654507 2065 tablet_replica.cc:442] T 7220f7e16dbb472b8a99b112b67ef940 P 529ae2624c744fd68bb2affef222bf5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:26:45.655244 2061 ts_tablet_manager.cc:1428] T 7220f7e16dbb472b8a99b112b67ef940 P 529ae2624c744fd68bb2affef222bf5a: Time spent starting tablet: real 0.023s user 0.007s sys 0.014s
I20260927 21:26:45.655588 2066 tablet_replica.cc:439] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } } }
I20260927 21:26:45.655923 2066 tablet_replica.cc:442] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:26:45.708961 2066 raft_consensus.cc:482] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:26:45.709216 2066 raft_consensus.cc:504] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } }
I20260927 21:26:45.710148 2066 leader_election.cc:288] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 529ae2624c744fd68bb2affef222bf5a (127.0.97.1:43365), 8e96fc7b50264245b05737d64a54e118 (127.0.97.3:42209)
I20260927 21:26:45.710743 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7220f7e16dbb472b8a99b112b67ef940" candidate_uuid: "06aca3f926d248c5a042cf7b5e67bc68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529ae2624c744fd68bb2affef222bf5a" is_pre_election: true
I20260927 21:26:45.710793 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7220f7e16dbb472b8a99b112b67ef940" candidate_uuid: "06aca3f926d248c5a042cf7b5e67bc68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e96fc7b50264245b05737d64a54e118" is_pre_election: true
I20260927 21:26:45.711086 1669 raft_consensus.cc:2454] T 7220f7e16dbb472b8a99b112b67ef940 P 529ae2624c744fd68bb2affef222bf5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06aca3f926d248c5a042cf7b5e67bc68 in term 0.
I20260927 21:26:45.711087 1948 raft_consensus.cc:2454] T 7220f7e16dbb472b8a99b112b67ef940 P 8e96fc7b50264245b05737d64a54e118 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06aca3f926d248c5a042cf7b5e67bc68 in term 0.
I20260927 21:26:45.711568 1740 leader_election.cc:302] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06aca3f926d248c5a042cf7b5e67bc68, 529ae2624c744fd68bb2affef222bf5a; no voters:
I20260927 21:26:45.711884 2066 raft_consensus.cc:2781] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:26:45.712075 2066 raft_consensus.cc:482] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:26:45.712198 2066 raft_consensus.cc:3037] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:45.714383 2066 raft_consensus.cc:504] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } }
I20260927 21:26:45.714977 2066 leader_election.cc:288] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68 [CANDIDATE]: Term 1 election: Requested vote from peers 529ae2624c744fd68bb2affef222bf5a (127.0.97.1:43365), 8e96fc7b50264245b05737d64a54e118 (127.0.97.3:42209)
I20260927 21:26:45.715566 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7220f7e16dbb472b8a99b112b67ef940" candidate_uuid: "06aca3f926d248c5a042cf7b5e67bc68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529ae2624c744fd68bb2affef222bf5a"
I20260927 21:26:45.715561 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7220f7e16dbb472b8a99b112b67ef940" candidate_uuid: "06aca3f926d248c5a042cf7b5e67bc68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e96fc7b50264245b05737d64a54e118"
I20260927 21:26:45.715793 1948 raft_consensus.cc:3037] T 7220f7e16dbb472b8a99b112b67ef940 P 8e96fc7b50264245b05737d64a54e118 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:45.715808 1669 raft_consensus.cc:3037] T 7220f7e16dbb472b8a99b112b67ef940 P 529ae2624c744fd68bb2affef222bf5a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:45.718956 1669 raft_consensus.cc:2454] T 7220f7e16dbb472b8a99b112b67ef940 P 529ae2624c744fd68bb2affef222bf5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06aca3f926d248c5a042cf7b5e67bc68 in term 1.
I20260927 21:26:45.718993 1948 raft_consensus.cc:2454] T 7220f7e16dbb472b8a99b112b67ef940 P 8e96fc7b50264245b05737d64a54e118 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06aca3f926d248c5a042cf7b5e67bc68 in term 1.
I20260927 21:26:45.719518 1740 leader_election.cc:302] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 06aca3f926d248c5a042cf7b5e67bc68, 529ae2624c744fd68bb2affef222bf5a, 8e96fc7b50264245b05737d64a54e118; no voters:
I20260927 21:26:45.719874 2066 raft_consensus.cc:2781] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:26:45.720314 2066 raft_consensus.cc:686] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68 [term 1 LEADER]: Becoming Leader. State: Replica: 06aca3f926d248c5a042cf7b5e67bc68, State: Running, Role: LEADER
I20260927 21:26:45.721002 2066 consensus_queue.cc:237] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } }
I20260927 21:26:45.722970 2067 tablet_replica.cc:439] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68: TxnStatusTablet state changed. Reason: New leader 06aca3f926d248c5a042cf7b5e67bc68. Latest consensus state: current_term: 1 leader_uuid: "06aca3f926d248c5a042cf7b5e67bc68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } } }
I20260927 21:26:45.723395 2067 tablet_replica.cc:442] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:26:45.724160 2069 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 21:26:45.724402 2069 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 21:26:45.724776 1512 catalog_manager.cc:5495] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06aca3f926d248c5a042cf7b5e67bc68 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "06aca3f926d248c5a042cf7b5e67bc68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } health_report { overall_health: UNKNOWN } } }
I20260927 21:26:45.761130 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:45.753601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35677 (local address 127.0.0.1:60964)
0927 21:26:45.753892 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:45.753914 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:45.754082 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:45.754768 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:45.754774 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:45.754788 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:26:45.754999 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:45.755011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.756689 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.756696 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:45.758305 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:45.758323 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.759714 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.759721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:45.759924 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:45.760008 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:26:45.760512 (+ 504us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:26:45.760536 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 21:26:45.760626 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 21:26:45.760779 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 21:26:45.761144 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:45.754104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60964 (local address 127.0.97.2:35677)
0927 21:26:45.754330 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:45.754338 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:45.754366 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:26:45.754502 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:45.754508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:45.754606 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:45.754718 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:26:45.755161 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.756515 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.758473 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.759516 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.759689 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:45.760213 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:26:45.760374 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:26:45.760499 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:45.760664 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 21:26:45.760762 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 21:26:45.766870 1948 raft_consensus.cc:1261] T 7220f7e16dbb472b8a99b112b67ef940 P 8e96fc7b50264245b05737d64a54e118 [term 1 FOLLOWER]: Refusing update from remote peer 06aca3f926d248c5a042cf7b5e67bc68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:26:45.766947 1669 raft_consensus.cc:1261] T 7220f7e16dbb472b8a99b112b67ef940 P 529ae2624c744fd68bb2affef222bf5a [term 1 FOLLOWER]: Refusing update from remote peer 06aca3f926d248c5a042cf7b5e67bc68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:26:45.767729 2067 consensus_queue.cc:1035] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:26:45.768039 2066 consensus_queue.cc:1035] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:26:45.787889 2024 tablet_replica.cc:439] T 7220f7e16dbb472b8a99b112b67ef940 P 8e96fc7b50264245b05737d64a54e118: TxnStatusTablet state changed. Reason: New leader 06aca3f926d248c5a042cf7b5e67bc68. Latest consensus state: current_term: 1 leader_uuid: "06aca3f926d248c5a042cf7b5e67bc68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } } }
I20260927 21:26:45.788228 2024 tablet_replica.cc:442] T 7220f7e16dbb472b8a99b112b67ef940 P 8e96fc7b50264245b05737d64a54e118: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:26:45.791778 2065 tablet_replica.cc:439] T 7220f7e16dbb472b8a99b112b67ef940 P 529ae2624c744fd68bb2affef222bf5a: TxnStatusTablet state changed. Reason: New leader 06aca3f926d248c5a042cf7b5e67bc68. Latest consensus state: current_term: 1 leader_uuid: "06aca3f926d248c5a042cf7b5e67bc68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } } }
I20260927 21:26:45.792068 2065 tablet_replica.cc:442] T 7220f7e16dbb472b8a99b112b67ef940 P 529ae2624c744fd68bb2affef222bf5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:26:45.792744 2072 mvcc.cc:205] Tried to move back new op lower bound from 7334069886013411328 to 7334069885836394496. Current Snapshot: MvccSnapshot[applied={T|T < 7334069886013411328}]
I20260927 21:26:45.799762 2073 mvcc.cc:205] Tried to move back new op lower bound from 7334069886013411328 to 7334069885836394496. Current Snapshot: MvccSnapshot[applied={T|T < 7334069886013411328}]
I20260927 21:26:45.802874 2067 tablet_replica.cc:439] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "06aca3f926d248c5a042cf7b5e67bc68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } } }
I20260927 21:26:45.803184 2067 tablet_replica.cc:442] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:26:45.805254 2024 tablet_replica.cc:439] T 7220f7e16dbb472b8a99b112b67ef940 P 8e96fc7b50264245b05737d64a54e118: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06aca3f926d248c5a042cf7b5e67bc68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } } }
I20260927 21:26:45.805573 2024 tablet_replica.cc:442] T 7220f7e16dbb472b8a99b112b67ef940 P 8e96fc7b50264245b05737d64a54e118: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:26:45.808775 2067 tablet_replica.cc:439] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "06aca3f926d248c5a042cf7b5e67bc68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } } }
I20260927 21:26:45.809039 2067 tablet_replica.cc:442] T 7220f7e16dbb472b8a99b112b67ef940 P 06aca3f926d248c5a042cf7b5e67bc68: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:26:45.810386 2074 mvcc.cc:205] Tried to move back new op lower bound from 7334069886013411328 to 7334069885836394496. Current Snapshot: MvccSnapshot[applied={T|T < 7334069886013411328}]
I20260927 21:26:45.812311 2065 tablet_replica.cc:439] T 7220f7e16dbb472b8a99b112b67ef940 P 529ae2624c744fd68bb2affef222bf5a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06aca3f926d248c5a042cf7b5e67bc68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 } } peers { permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 } } peers { permanent_uuid: "8e96fc7b50264245b05737d64a54e118" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42209 } } }
I20260927 21:26:45.812536 2065 tablet_replica.cc:442] T 7220f7e16dbb472b8a99b112b67ef940 P 529ae2624c744fd68bb2affef222bf5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:26:45.833142 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:45.819584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41078 (local address 127.0.97.3:42209)
0927 21:26:45.825498 (+ 5914us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:45.825505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:45.825955 (+ 450us) server_negotiation.cc:407] Connection header received
0927 21:26:45.826152 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:45.826159 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:45.826271 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:45.826473 (+ 202us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:26:45.827021 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.828200 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.829805 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.830907 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.831041 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:45.831623 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:26:45.832388 (+ 765us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:26:45.832537 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:45.832641 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 21:26:45.832736 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5753,"thread_start_us":5568,"threads_started":1}
I20260927 21:26:45.849786 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:45.843812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33091 (local address 127.0.97.2:35677)
0927 21:26:45.844018 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:45.844024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:45.844909 (+ 885us) server_negotiation.cc:407] Connection header received
0927 21:26:45.845095 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:45.845099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:45.845163 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:45.845257 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:26:45.845811 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.846626 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.847957 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.848405 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.848547 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:45.848959 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:26:45.849100 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:26:45.849199 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:45.849336 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 21:26:45.849416 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 21:26:45.849844 2085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:45.843679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35677 (local address 127.0.97.3:33091)
0927 21:26:45.844802 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:45.844818 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:45.844942 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:45.845283 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:45.845287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:45.845299 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:26:45.845651 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:45.845663 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.846756 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.846762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:45.847807 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:45.847818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.848581 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.848589 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:45.848725 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:45.848782 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:26:45.849199 (+ 417us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:26:45.849219 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 21:26:45.849314 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 21:26:45.849446 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"thread_start_us":813,"threads_started":1}
I20260927 21:26:45.860538 1669 raft_consensus.cc:1261] T 93fe9dcfbb034497969f19785ea72cbe P 529ae2624c744fd68bb2affef222bf5a [term 1 FOLLOWER]: Refusing update from remote peer 8e96fc7b50264245b05737d64a54e118: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:26:45.860672 1814 raft_consensus.cc:1261] T 93fe9dcfbb034497969f19785ea72cbe P 06aca3f926d248c5a042cf7b5e67bc68 [term 1 FOLLOWER]: Refusing update from remote peer 8e96fc7b50264245b05737d64a54e118: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:26:45.861325 2050 consensus_queue.cc:1035] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118 [LEADER]: Connected to new peer: Peer: permanent_uuid: "529ae2624c744fd68bb2affef222bf5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:26:45.861759 2024 consensus_queue.cc:1035] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06aca3f926d248c5a042cf7b5e67bc68" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:26:45.910748 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:45.893506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58549 (local address 127.0.97.3:42209)
0927 21:26:45.893715 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:45.893722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:45.900256 (+ 6534us) server_negotiation.cc:407] Connection header received
0927 21:26:45.900913 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:45.900918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:45.901016 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:45.901068 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:26:45.902680 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.903776 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.908527 (+ 4751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.909063 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.909197 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:45.909986 (+ 789us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:26:45.910118 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:26:45.910205 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:45.910376 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 21:26:45.910457 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 21:26:45.911147 2090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:45.893057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42209 (local address 127.0.97.2:58549)
0927 21:26:45.900153 (+ 7096us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:45.900174 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:45.900724 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:45.901543 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:45.901546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:45.901561 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:26:45.902502 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:45.902516 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.907132 (+ 4616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.907136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:45.908351 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:45.908363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.909346 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.909353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:45.909532 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:45.909652 (+ 120us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:26:45.910217 (+ 565us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:26:45.910242 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 21:26:45.910322 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 21:26:45.910499 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6252,"thread_start_us":6126,"threads_started":1}
I20260927 21:26:45.950927 1513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:26:45.951223 1513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:26:45.955052 1513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a2333e800441a6a0983e7b07b47387: Sending DeleteTablet for 3 replicas of tablet 93fe9dcfbb034497969f19785ea72cbe
I20260927 21:26:45.956123 1650 tablet_service.cc:1514] Processing DeleteTablet for tablet 93fe9dcfbb034497969f19785ea72cbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:26:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42752
I20260927 21:26:45.956924 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet 93fe9dcfbb034497969f19785ea72cbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:26:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60942
I20260927 21:26:45.957489 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 93fe9dcfbb034497969f19785ea72cbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:26:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41064
I20260927 21:26:45.965191 2096 tablet_replica.cc:330] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118: stopping tablet replica
I20260927 21:26:45.965480 2096 raft_consensus.cc:2229] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:26:45.966228 2096 raft_consensus.cc:2258] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:26:45.967733 2096 ts_tablet_manager.cc:1905] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:26:45.983946 2094 tablet_replica.cc:330] T 93fe9dcfbb034497969f19785ea72cbe P 06aca3f926d248c5a042cf7b5e67bc68: stopping tablet replica
I20260927 21:26:45.984198 2094 raft_consensus.cc:2229] T 93fe9dcfbb034497969f19785ea72cbe P 06aca3f926d248c5a042cf7b5e67bc68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:26:45.984519 2094 raft_consensus.cc:2258] T 93fe9dcfbb034497969f19785ea72cbe P 06aca3f926d248c5a042cf7b5e67bc68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:26:45.985155 2093 tablet_replica.cc:330] T 93fe9dcfbb034497969f19785ea72cbe P 529ae2624c744fd68bb2affef222bf5a: stopping tablet replica
I20260927 21:26:45.985432 2093 raft_consensus.cc:2229] T 93fe9dcfbb034497969f19785ea72cbe P 529ae2624c744fd68bb2affef222bf5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:26:45.985894 2094 ts_tablet_manager.cc:1905] T 93fe9dcfbb034497969f19785ea72cbe P 06aca3f926d248c5a042cf7b5e67bc68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:26:45.986227 2093 raft_consensus.cc:2258] T 93fe9dcfbb034497969f19785ea72cbe P 529ae2624c744fd68bb2affef222bf5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:26:45.989015 2096 ts_tablet_manager.cc:1918] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:26:45.989187 2093 ts_tablet_manager.cc:1905] T 93fe9dcfbb034497969f19785ea72cbe P 529ae2624c744fd68bb2affef222bf5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:26:45.989806 2096 log.cc:1196] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-2/wal/wals/93fe9dcfbb034497969f19785ea72cbe
I20260927 21:26:45.990424 2096 ts_tablet_manager.cc:1939] T 93fe9dcfbb034497969f19785ea72cbe P 8e96fc7b50264245b05737d64a54e118: Deleting consensus metadata
I20260927 21:26:45.991707 1498 catalog_manager.cc:4841] TS 8e96fc7b50264245b05737d64a54e118 (127.0.97.3:42209): tablet 93fe9dcfbb034497969f19785ea72cbe (table test-table [id=970d6e2052d24131886c7ca3fec7958b]) successfully deleted
I20260927 21:26:45.993434 2094 ts_tablet_manager.cc:1918] T 93fe9dcfbb034497969f19785ea72cbe P 06aca3f926d248c5a042cf7b5e67bc68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:26:45.993568 2094 log.cc:1196] T 93fe9dcfbb034497969f19785ea72cbe P 06aca3f926d248c5a042cf7b5e67bc68: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-1/wal/wals/93fe9dcfbb034497969f19785ea72cbe
I20260927 21:26:45.994086 2094 ts_tablet_manager.cc:1939] T 93fe9dcfbb034497969f19785ea72cbe P 06aca3f926d248c5a042cf7b5e67bc68: Deleting consensus metadata
I20260927 21:26:45.995265 1498 catalog_manager.cc:4841] TS 06aca3f926d248c5a042cf7b5e67bc68 (127.0.97.2:35677): tablet 93fe9dcfbb034497969f19785ea72cbe (table test-table [id=970d6e2052d24131886c7ca3fec7958b]) successfully deleted
I20260927 21:26:46.001087 2093 ts_tablet_manager.cc:1918] T 93fe9dcfbb034497969f19785ea72cbe P 529ae2624c744fd68bb2affef222bf5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:26:46.001324 2093 log.cc:1196] T 93fe9dcfbb034497969f19785ea72cbe P 529ae2624c744fd68bb2affef222bf5a: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790544390748963-388-0/minicluster-data/ts-0/wal/wals/93fe9dcfbb034497969f19785ea72cbe
I20260927 21:26:46.001876 2093 ts_tablet_manager.cc:1939] T 93fe9dcfbb034497969f19785ea72cbe P 529ae2624c744fd68bb2affef222bf5a: Deleting consensus metadata
Sep 27 21:26:45 dist-test-slave-bxm0 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544404, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 21:26:46.003118 1497 catalog_manager.cc:4841] TS 529ae2624c744fd68bb2affef222bf5a (127.0.97.1:43365): tablet 93fe9dcfbb034497969f19785ea72cbe (table test-table [id=970d6e2052d24131886c7ca3fec7958b]) successfully deleted
I20260927 21:26:46.012701 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:45.971092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42768 (local address 127.0.97.1:43365)
0927 21:26:45.971326 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:45.971335 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:45.978606 (+ 7271us) server_negotiation.cc:407] Connection header received
0927 21:26:45.978839 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:45.978845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:45.978969 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:45.979078 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:45.993828 (+ 14750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.994924 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.996962 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:45.997766 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:45.997899 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:46.004173 (+ 6274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:46.004191 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:46.004216 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:46.004262 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:46.008723 (+ 4461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:46.009719 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:46.009730 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:46.009734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:46.009892 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:46.010317 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:46.010322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:46.010326 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:46.010751 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:46.011020 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:46.012068 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0927 21:26:46.012322 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 21:26:46.013540 1715 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42768
I20260927 21:26:46.039646 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:46.017375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57334 (local address 127.0.97.62:40507)
0927 21:26:46.017671 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:46.017679 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:46.022550 (+ 4871us) server_negotiation.cc:407] Connection header received
0927 21:26:46.022643 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:46.022649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:46.022736 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:46.022854 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:46.024643 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:46.025875 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:46.027693 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:46.028382 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:46.028529 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:46.031663 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:46.031670 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:46.031675 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:46.031720 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:46.035854 (+ 4134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:46.036744 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:46.036749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:46.036753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:46.036890 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:46.037300 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:46.037305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:46.037308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:46.037695 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:46.037926 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:46.039053 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0927 21:26:46.039286 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
W20260927 21:26:46.040542 1513 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57334
I20260927 21:26:46.043852 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 1584
I20260927 21:26:46.063305 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:46.044680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42780 (local address 127.0.97.1:43365)
0927 21:26:46.045350 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:46.045357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:46.045380 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:26:46.045703 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:46.045709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:46.045818 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:46.045896 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:46.047730 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:46.048840 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:46.050756 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:46.051422 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:46.051527 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:46.055351 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:46.055359 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:46.055364 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:46.055414 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:46.059078 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:46.060211 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:46.060217 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:46.060221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:46.060415 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:46.060998 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:46.061003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:46.061007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:46.061866 (+ 859us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:46.062130 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:46.062689 (+ 559us) server_negotiation.cc:299] Negotiation successful
0927 21:26:46.062940 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524}
W20260927 21:26:46.064723 1715 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42780
W20260927 21:26:46.065340 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 1584: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:26:46.065661 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 1584
I20260927 21:26:46.089952 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 1724
Sep 27 21:26:46 dist-test-slave-bxm0 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544404, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 21:26:46.110719 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:46.090730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60976 (local address 127.0.97.2:35677)
0927 21:26:46.090917 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:46.090923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:46.091070 (+ 147us) server_negotiation.cc:407] Connection header received
0927 21:26:46.091234 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:46.091240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:46.091336 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:46.091464 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:46.093060 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:46.094235 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:46.096083 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:46.096754 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:46.096919 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:46.102702 (+ 5783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:46.102722 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:46.102745 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:46.102803 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:46.107196 (+ 4393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:46.108220 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:46.108237 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:46.108241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:46.108423 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:46.108855 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:46.108861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:46.108865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:46.109381 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:46.109809 (+ 428us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:46.110083 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 21:26:46.110331 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 21:26:46.111480 1854 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60976
W20260927 21:26:46.112241 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 1724: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:26:46.112438 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 1724
I20260927 21:26:46.136317 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 1863
Sep 27 21:26:46 dist-test-slave-bxm0 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544404, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 21:26:46.154675 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:46.137052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41094 (local address 127.0.97.3:42209)
0927 21:26:46.137232 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:46.137240 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:46.137318 (+ 78us) server_negotiation.cc:407] Connection header received
0927 21:26:46.137482 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:46.137495 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:46.137577 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:46.137717 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:46.139257 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:46.140017 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:46.141895 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:46.142395 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:46.142516 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:46.147669 (+ 5153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:46.147684 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:46.147701 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:46.147749 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:46.151222 (+ 3473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:46.152028 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:46.152037 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:46.152040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:46.152178 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:46.152609 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:46.152616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:46.152619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:46.153146 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:46.153569 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:46.154052 (+ 483us) server_negotiation.cc:299] Negotiation successful
0927 21:26:46.154292 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 21:26:46.155433 1993 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41094
W20260927 21:26:46.156147 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 1863: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:26:46.156324 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 1863
I20260927 21:26:46.174928 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 1481
I20260927 21:26:46.176120 1553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:26:46.293948 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 1481
2026-09-27T21:26:46Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5003 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:26:46 dist-test-slave-bxm0 krb5kdc[2104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:26:46 dist-test-slave-bxm0 krb5kdc[2104](info): set up 2 sockets
Sep 27 21:26:46 dist-test-slave-bxm0 krb5kdc[2104](info): commencing operation
krb5kdc: starting...
W20260927 21:26:47.373870 388 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:26:47 dist-test-slave-bxm0 krb5kdc[2104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:26:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:26:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:26:47.565927 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37339
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:26:47.900790 2117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:26:47.901199 2117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:26:47.901378 2117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:26:47.909137 2117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:26:47.909260 2117 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 21:26:47.909327 2117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:26:47.909415 2117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:26:47.909498 2117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:26:47.909601 2117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:26:47.921066 2117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37339
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:26:47.922569 2117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:26:47.924059 2117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:26:47.937279 2122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:47.937332 2123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:47.937512 2125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:26:47.938365 2117 server_base.cc:1034] running on GCE node
I20260927 21:26:47.939190 2117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:26:47.940574 2117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:26:47.941833 2117 hybrid_clock.cc:648] HybridClock initialized: now 1790544407941768 us; error 45 us; skew 500 ppm
I20260927 21:26:47.942175 2117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:26:47 dist-test-slave-bxm0 krb5kdc[2104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:26:47.945524 2117 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:26:47.947707 2117 webserver.cc:460] Webserver started at http://127.0.97.62:45035/ using document root <none> and password file <none>
I20260927 21:26:47.948824 2117 fs_manager.cc:359] Metadata directory not provided
I20260927 21:26:47.948947 2117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:26:47.949257 2117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:26:47.952636 2117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "3be07d0d94614d2891dc897eec726613"
format_stamp: "Formatted at 2026-09-27 21:26:47 on dist-test-slave-bxm0"
server_key: "0fb459853b26639b8a5daf0970cf4ecf"
server_key_iv: "f8523aba186dc6ba562d543b08c7f069"
server_key_version: "encryptionkey@0"
I20260927 21:26:47.953470 2117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "3be07d0d94614d2891dc897eec726613"
format_stamp: "Formatted at 2026-09-27 21:26:47 on dist-test-slave-bxm0"
server_key: "0fb459853b26639b8a5daf0970cf4ecf"
server_key_iv: "f8523aba186dc6ba562d543b08c7f069"
server_key_version: "encryptionkey@0"
I20260927 21:26:47.959435 2117 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260927 21:26:47.963287 2132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:47.964818 2117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:26:47.965044 2117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "3be07d0d94614d2891dc897eec726613"
format_stamp: "Formatted at 2026-09-27 21:26:47 on dist-test-slave-bxm0"
server_key: "0fb459853b26639b8a5daf0970cf4ecf"
server_key_iv: "f8523aba186dc6ba562d543b08c7f069"
server_key_version: "encryptionkey@0"
I20260927 21:26:47.965310 2117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:26:47.997639 2117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:26:48.002614 2117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:26:48.002910 2117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:26:48.024631 2117 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37339
I20260927 21:26:48.024654 2193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37339 every 8 connection(s)
I20260927 21:26:48.026130 2117 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:26:48.029949 2194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:48.034667 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 2117
I20260927 21:26:48.034893 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:26:48.035683 388 external_mini_cluster.cc:1428] Setting key 259e73af110c49b1a07785235ae564e5
I20260927 21:26:48.043118 2194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be07d0d94614d2891dc897eec726613: Bootstrap starting.
I20260927 21:26:48.047345 2194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3be07d0d94614d2891dc897eec726613: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:48.048945 2194 log.cc:824] T 00000000000000000000000000000000 P 3be07d0d94614d2891dc897eec726613: Log is configured to *not* fsync() on all Append() calls
Sep 27 21:26:48 dist-test-slave-bxm0 krb5kdc[2104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:26:48.052470 2194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be07d0d94614d2891dc897eec726613: No bootstrap required, opened a new log
I20260927 21:26:48.061106 2194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3be07d0d94614d2891dc897eec726613 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be07d0d94614d2891dc897eec726613" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37339 } }
I20260927 21:26:48.061568 2194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3be07d0d94614d2891dc897eec726613 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:48.061663 2197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:48.038027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41026 (local address 127.0.97.62:37339)
0927 21:26:48.038896 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:48.038911 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:48.038952 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:26:48.041135 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:48.041162 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:48.041791 (+ 629us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:48.042197 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:48.044200 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:48.045600 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:48.047739 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:48.048609 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:48.048750 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:48.053424 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:48.053433 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:48.053450 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:48.053494 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:48.057331 (+ 3837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:48.058174 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:48.058181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:48.058188 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:48.058361 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:48.058780 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:48.058784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:48.058786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:48.059351 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:48.059676 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:48.060155 (+ 479us) server_negotiation.cc:299] Negotiation successful
0927 21:26:48.060580 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":441,"threads_started":1}
I20260927 21:26:48.061767 2194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3be07d0d94614d2891dc897eec726613 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3be07d0d94614d2891dc897eec726613, State: Initialized, Role: FOLLOWER
I20260927 21:26:48.062466 2194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3be07d0d94614d2891dc897eec726613 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be07d0d94614d2891dc897eec726613" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37339 } }
I20260927 21:26:48.062815 2194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3be07d0d94614d2891dc897eec726613 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:26:48.062995 2194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3be07d0d94614d2891dc897eec726613 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:26:48.063202 2194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3be07d0d94614d2891dc897eec726613 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:48.066870 2194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3be07d0d94614d2891dc897eec726613 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be07d0d94614d2891dc897eec726613" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37339 } }
I20260927 21:26:48.067657 2194 leader_election.cc:302] T 00000000000000000000000000000000 P 3be07d0d94614d2891dc897eec726613 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3be07d0d94614d2891dc897eec726613; no voters:
I20260927 21:26:48.068467 2194 leader_election.cc:288] T 00000000000000000000000000000000 P 3be07d0d94614d2891dc897eec726613 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:26:48.068624 2199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3be07d0d94614d2891dc897eec726613 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:26:48.070960 2199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3be07d0d94614d2891dc897eec726613 [term 1 LEADER]: Becoming Leader. State: Replica: 3be07d0d94614d2891dc897eec726613, State: Running, Role: LEADER
I20260927 21:26:48.071130 2194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3be07d0d94614d2891dc897eec726613 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:26:48.071707 2199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3be07d0d94614d2891dc897eec726613 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be07d0d94614d2891dc897eec726613" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37339 } }
I20260927 21:26:48.075210 2201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be07d0d94614d2891dc897eec726613 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3be07d0d94614d2891dc897eec726613. Latest consensus state: current_term: 1 leader_uuid: "3be07d0d94614d2891dc897eec726613" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be07d0d94614d2891dc897eec726613" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37339 } } }
I20260927 21:26:48.075610 2201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be07d0d94614d2891dc897eec726613 [sys.catalog]: This master's current role is: LEADER
I20260927 21:26:48.075338 2200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be07d0d94614d2891dc897eec726613 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3be07d0d94614d2891dc897eec726613" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be07d0d94614d2891dc897eec726613" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37339 } } }
I20260927 21:26:48.075776 2200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be07d0d94614d2891dc897eec726613 [sys.catalog]: This master's current role is: LEADER
I20260927 21:26:48.077219 2208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:26:48.082163 2208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:26:48.091125 2208 catalog_manager.cc:1347] Generated new cluster ID: 783c619ae3c0403db8d9ee0417897a4e
I20260927 21:26:48.091253 2208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:26:48.107668 2208 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:26:48.109699 2208 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:26:48.122260 2208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3be07d0d94614d2891dc897eec726613: Generated new TSK 0
I20260927 21:26:48.123409 2208 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 21:26:48.185789 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37339
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:26:48.537899 2220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:26:48.538231 2220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:26:48.538419 2220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:26:48.545533 2220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:26:48.545709 2220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:26:48.545879 2220 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 21:26:48.545969 2220 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:26:48.546082 2220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:26:48.555918 2220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:26:48.557286 2220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:26:48.558621 2220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:26:48.572034 2225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:48.572070 2228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:48.572546 2226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:26:48.573266 2220 server_base.cc:1034] running on GCE node
I20260927 21:26:48.573951 2220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:26:48.574916 2220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:26:48.576143 2220 hybrid_clock.cc:648] HybridClock initialized: now 1790544408576085 us; error 46 us; skew 500 ppm
I20260927 21:26:48.576551 2220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:26:48 dist-test-slave-bxm0 krb5kdc[2104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:26:48.579860 2220 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 21:26:48.581420 2220 webserver.cc:460] Webserver started at http://127.0.97.1:37039/ using document root <none> and password file <none>
I20260927 21:26:48.582166 2220 fs_manager.cc:359] Metadata directory not provided
I20260927 21:26:48.582273 2220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:26:48.582561 2220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:26:48.585558 2220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-0/data/instance:
uuid: "0a4fb22e29ef4818bf36fa4aca79c34e"
format_stamp: "Formatted at 2026-09-27 21:26:48 on dist-test-slave-bxm0"
server_key: "53dbff83da5522855baa3e01ad44b68a"
server_key_iv: "5c26a15f5bf14fb120904919679b87bb"
server_key_version: "encryptionkey@0"
I20260927 21:26:48.586460 2220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0a4fb22e29ef4818bf36fa4aca79c34e"
format_stamp: "Formatted at 2026-09-27 21:26:48 on dist-test-slave-bxm0"
server_key: "53dbff83da5522855baa3e01ad44b68a"
server_key_iv: "5c26a15f5bf14fb120904919679b87bb"
server_key_version: "encryptionkey@0"
I20260927 21:26:48.591711 2220 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:26:48.595326 2235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:48.596751 2220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:26:48.596937 2220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "0a4fb22e29ef4818bf36fa4aca79c34e"
format_stamp: "Formatted at 2026-09-27 21:26:48 on dist-test-slave-bxm0"
server_key: "53dbff83da5522855baa3e01ad44b68a"
server_key_iv: "5c26a15f5bf14fb120904919679b87bb"
server_key_version: "encryptionkey@0"
I20260927 21:26:48.597146 2220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:26:48.623993 2220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:26:48.628353 2220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:26:48.628665 2220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:26:48.631460 2220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:26:48.631609 2220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:48.631739 2220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:26:48.631855 2220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:26:48 dist-test-slave-bxm0 krb5kdc[2104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:26:48.656888 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:48.631956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45515 (local address 127.0.97.62:37339)
0927 21:26:48.632737 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:48.632743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:48.633902 (+ 1159us) server_negotiation.cc:407] Connection header received
0927 21:26:48.636116 (+ 2214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:48.636122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:48.636218 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:48.636368 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:48.639647 (+ 3279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:48.640641 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:48.643948 (+ 3307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:48.644519 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:48.644676 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:48.649871 (+ 5195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:48.649877 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:48.649880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:48.649914 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:48.653259 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:48.654442 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:48.654447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:48.654451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:48.654585 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:48.655048 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:48.655053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:48.655057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:48.655325 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:48.655480 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:48.656307 (+ 827us) server_negotiation.cc:299] Negotiation successful
0927 21:26:48.656526 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":538,"threads_started":1}
I20260927 21:26:48.658124 2245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:48.632729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37339 (local address 127.0.97.1:45515)
0927 21:26:48.633710 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:48.633763 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:48.635802 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:48.636968 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:48.636989 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:48.638148 (+ 1159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:26:48.639367 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:48.639403 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:48.640832 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:48.640840 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:48.643766 (+ 2926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:48.643787 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:48.644828 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:48.644839 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:48.645244 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:48.646217 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:26:48.646255 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:26:48.649536 (+ 3281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:26:48.653431 (+ 3895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:48.653441 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:48.653464 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:48.654262 (+ 798us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:48.654713 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:48.654716 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:48.654719 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:48.654919 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:48.655459 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:26:48.655467 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:26:48.655923 (+ 456us) client_negotiation.cc:769] Sending connection context
0927 21:26:48.656200 (+ 277us) client_negotiation.cc:240] Negotiation successful
0927 21:26:48.656774 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":423,"threads_started":1}
I20260927 21:26:48.679708 2308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:48.663919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37339 (local address 127.0.97.1:38741)
0927 21:26:48.664777 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:48.664803 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:48.664983 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:48.665329 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:48.665334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:48.665978 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:26:48.666395 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:48.666406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:48.667699 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:48.667703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:48.669590 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:48.669603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:48.670693 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:48.670707 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:48.670999 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:48.671844 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:26:48.671877 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:26:48.673311 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:26:48.676641 (+ 3330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:48.676648 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:48.676653 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:48.677056 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:48.677461 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:48.677467 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:48.677470 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:48.677603 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:48.678329 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:26:48.678337 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:26:48.678523 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 21:26:48.678708 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 21:26:48.679110 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":474,"threads_started":1}
I20260927 21:26:48.680805 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:48.663871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38741 (local address 127.0.97.62:37339)
0927 21:26:48.664071 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:48.664080 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:48.664900 (+ 820us) server_negotiation.cc:407] Connection header received
0927 21:26:48.665116 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:48.665121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:48.665194 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:48.665297 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:48.666548 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:48.667524 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:48.669803 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:48.670431 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:48.670615 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:48.673524 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:48.673530 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:48.673534 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:48.673578 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:48.676501 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:48.677187 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:48.677197 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:48.677200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:48.677358 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:48.677754 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:48.677760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:48.677763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:48.678166 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:48.680002 (+ 1836us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:48.680188 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 21:26:48.680448 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:26:48.686532 2220 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42437
I20260927 21:26:48.686616 2355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42437 every 8 connection(s)
I20260927 21:26:48.687788 2220 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:26:48.689962 2356 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37339
I20260927 21:26:48.690178 2356 heartbeater.cc:464] Registering TS with master...
I20260927 21:26:48.690760 2356 heartbeater.cc:511] Master 127.0.97.62:37339 requested a full tablet report, sending...
I20260927 21:26:48.692648 2149 ts_manager.cc:194] Registered new tserver with Master: 0a4fb22e29ef4818bf36fa4aca79c34e (127.0.97.1:42437)
I20260927 21:26:48.694164 2149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45515
I20260927 21:26:48.696882 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 2220
I20260927 21:26:48.697121 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-0/wal/instance
I20260927 21:26:48.697778 388 external_mini_cluster.cc:1428] Setting key 79f1d5a9f07f08af7180142b876e9ca0
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 21:26:48.738977 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37339
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:26:49.062402 2360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:26:49.062691 2360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:26:49.062856 2360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:26:49.070300 2360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:26:49.070478 2360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:26:49.070649 2360 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 21:26:49.070727 2360 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:26:49.070832 2360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:26:49.080907 2360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:26:49.082304 2360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:26:49.083630 2360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:26:49.096942 2365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:49.096853 2366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:49.096897 2368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:26:49.097919 2360 server_base.cc:1034] running on GCE node
I20260927 21:26:49.098531 2360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:26:49.099516 2360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:26:49.100761 2360 hybrid_clock.cc:648] HybridClock initialized: now 1790544409100698 us; error 58 us; skew 500 ppm
I20260927 21:26:49.101166 2360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:26:49 dist-test-slave-bxm0 krb5kdc[2104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:26:49.104600 2360 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 21:26:49.106307 2360 webserver.cc:460] Webserver started at http://127.0.97.2:42215/ using document root <none> and password file <none>
I20260927 21:26:49.107054 2360 fs_manager.cc:359] Metadata directory not provided
I20260927 21:26:49.107211 2360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:26:49.107550 2360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:26:49.110754 2360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-1/data/instance:
uuid: "df4ce836ef04452a9334f9e3cb62ed5f"
format_stamp: "Formatted at 2026-09-27 21:26:49 on dist-test-slave-bxm0"
server_key: "2948a1e83da0aac4e89ef4ba9a61ad14"
server_key_iv: "e2e757d15253d218daeefda21d16e78a"
server_key_version: "encryptionkey@0"
I20260927 21:26:49.111709 2360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-1/wal/instance:
uuid: "df4ce836ef04452a9334f9e3cb62ed5f"
format_stamp: "Formatted at 2026-09-27 21:26:49 on dist-test-slave-bxm0"
server_key: "2948a1e83da0aac4e89ef4ba9a61ad14"
server_key_iv: "e2e757d15253d218daeefda21d16e78a"
server_key_version: "encryptionkey@0"
I20260927 21:26:49.117278 2360 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 21:26:49.121140 2375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:49.122684 2360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:26:49.122880 2360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "df4ce836ef04452a9334f9e3cb62ed5f"
format_stamp: "Formatted at 2026-09-27 21:26:49 on dist-test-slave-bxm0"
server_key: "2948a1e83da0aac4e89ef4ba9a61ad14"
server_key_iv: "e2e757d15253d218daeefda21d16e78a"
server_key_version: "encryptionkey@0"
I20260927 21:26:49.123104 2360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:26:49.138795 2360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:26:49.142804 2360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:26:49.143090 2360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:26:49.145823 2360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:26:49.145943 2360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:49.146082 2360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:26:49.146165 2360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:26:49 dist-test-slave-bxm0 krb5kdc[2104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:26:49.171778 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:49.146297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40447 (local address 127.0.97.62:37339)
0927 21:26:49.146486 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:49.146494 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:49.147900 (+ 1406us) server_negotiation.cc:407] Connection header received
0927 21:26:49.150056 (+ 2156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:49.150061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:49.150166 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:49.150279 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:49.152493 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.153604 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.156967 (+ 3363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.157566 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.157760 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:49.163490 (+ 5730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:49.163495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:49.163498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:49.163579 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:49.167709 (+ 4130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:49.168587 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:49.168595 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:49.168601 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:49.168779 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:49.169290 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:49.169297 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:49.169303 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:49.169840 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:49.169993 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:49.170933 (+ 940us) server_negotiation.cc:299] Negotiation successful
0927 21:26:49.171243 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 21:26:49.172748 2384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:49.146933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37339 (local address 127.0.97.2:40447)
0927 21:26:49.147733 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:49.147772 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:49.149757 (+ 1985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:49.150525 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:49.150533 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:49.151291 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:26:49.152278 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:49.152307 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.153822 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.153829 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:49.156702 (+ 2873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:49.156721 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.157886 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.157897 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:49.158416 (+ 519us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:49.159329 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:26:49.159370 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:26:49.163110 (+ 3740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:26:49.167925 (+ 4815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:49.167938 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:49.167983 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:49.168452 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:49.168899 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:49.168902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:49.168904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:49.169168 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:49.170016 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:26:49.170027 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:26:49.170460 (+ 433us) client_negotiation.cc:769] Sending connection context
0927 21:26:49.170781 (+ 321us) client_negotiation.cc:240] Negotiation successful
0927 21:26:49.171299 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":359,"threads_started":1}
I20260927 21:26:49.195730 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:49.179598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48287 (local address 127.0.97.62:37339)
0927 21:26:49.180161 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:49.180169 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:49.180508 (+ 339us) server_negotiation.cc:407] Connection header received
0927 21:26:49.180709 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:49.180715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:49.180790 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:49.180890 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:49.181855 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.182938 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.184648 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.185069 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.185213 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:49.187838 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:49.187848 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:49.187856 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:49.187907 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:49.192425 (+ 4518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:49.193491 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:49.193499 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:49.193503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:49.193714 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:49.194096 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:49.194101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:49.194104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:49.194568 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:49.194719 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:49.195132 (+ 413us) server_negotiation.cc:299] Negotiation successful
0927 21:26:49.195371 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373}
I20260927 21:26:49.195838 2443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:49.179676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37339 (local address 127.0.97.2:48287)
0927 21:26:49.180392 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:49.180408 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:49.180587 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:49.180914 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:49.180918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:49.181373 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:26:49.181718 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:49.181729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.183081 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.183089 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:49.184476 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:49.184488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.185260 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.185266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:49.185428 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:49.186319 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:26:49.186356 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:26:49.187620 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:26:49.192649 (+ 5029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:49.192699 (+ 50us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:49.192706 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:49.193326 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:49.193844 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:49.193853 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:49.193856 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:49.193975 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:49.194693 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:26:49.194696 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:26:49.194866 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 21:26:49.195027 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 21:26:49.195401 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":487,"threads_started":1}
I20260927 21:26:49.204123 2360 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35833
I20260927 21:26:49.204172 2494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35833 every 8 connection(s)
I20260927 21:26:49.205371 2360 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:26:49.207481 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 2360
I20260927 21:26:49.207587 2495 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37339
I20260927 21:26:49.207677 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-1/wal/instance
I20260927 21:26:49.207831 2495 heartbeater.cc:464] Registering TS with master...
I20260927 21:26:49.208222 388 external_mini_cluster.cc:1428] Setting key 03628bc2178a80eec2b4de90b04b873e
I20260927 21:26:49.208657 2495 heartbeater.cc:511] Master 127.0.97.62:37339 requested a full tablet report, sending...
I20260927 21:26:49.210114 2148 ts_manager.cc:194] Registered new tserver with Master: df4ce836ef04452a9334f9e3cb62ed5f (127.0.97.2:35833)
I20260927 21:26:49.211030 2148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40447
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 21:26:49.252853 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37339
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:26:49.589771 2499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:26:49.590083 2499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:26:49.590196 2499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:26:49.597906 2499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:26:49.598037 2499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:26:49.598131 2499 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 21:26:49.598191 2499 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:26:49.598266 2499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:26:49.608628 2499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:26:49.609970 2499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:26:49.611330 2499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:26:49.626390 2504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:49.626340 2505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:26:49.627270 2499 server_base.cc:1034] running on GCE node
W20260927 21:26:49.628351 2507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:26:49.629200 2499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:26:49.630730 2499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:26:49.631989 2499 hybrid_clock.cc:648] HybridClock initialized: now 1790544409631943 us; error 46 us; skew 500 ppm
I20260927 21:26:49.632368 2499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:26:49 dist-test-slave-bxm0 krb5kdc[2104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:26:49.635757 2499 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 21:26:49.637410 2499 webserver.cc:460] Webserver started at http://127.0.97.3:36703/ using document root <none> and password file <none>
I20260927 21:26:49.638204 2499 fs_manager.cc:359] Metadata directory not provided
I20260927 21:26:49.638356 2499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:26:49.638681 2499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:26:49.641675 2499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-2/data/instance:
uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8"
format_stamp: "Formatted at 2026-09-27 21:26:49 on dist-test-slave-bxm0"
server_key: "0c601ebb5e53f0ae0818df69432a45f3"
server_key_iv: "2aa9020dec93dc1edcc76adbf395aab6"
server_key_version: "encryptionkey@0"
I20260927 21:26:49.642485 2499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-2/wal/instance:
uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8"
format_stamp: "Formatted at 2026-09-27 21:26:49 on dist-test-slave-bxm0"
server_key: "0c601ebb5e53f0ae0818df69432a45f3"
server_key_iv: "2aa9020dec93dc1edcc76adbf395aab6"
server_key_version: "encryptionkey@0"
I20260927 21:26:49.647667 2499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:26:49.651258 2514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:49.652738 2499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:26:49.652899 2499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8"
format_stamp: "Formatted at 2026-09-27 21:26:49 on dist-test-slave-bxm0"
server_key: "0c601ebb5e53f0ae0818df69432a45f3"
server_key_iv: "2aa9020dec93dc1edcc76adbf395aab6"
server_key_version: "encryptionkey@0"
I20260927 21:26:49.653087 2499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:26:49.671772 2499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:26:49.676424 2499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:26:49.676728 2499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:26:49.679428 2499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:26:49.679585 2499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:49.679731 2499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:26:49.679826 2499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:26:49 dist-test-slave-bxm0 krb5kdc[2104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:26:49.697017 2356 heartbeater.cc:503] Master 127.0.97.62:37339 was elected leader, sending a full tablet report...
I20260927 21:26:49.704663 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:49.680246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48889 (local address 127.0.97.62:37339)
0927 21:26:49.680439 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:49.680445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:49.681972 (+ 1527us) server_negotiation.cc:407] Connection header received
0927 21:26:49.684706 (+ 2734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:49.684717 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:49.684840 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:49.684961 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:49.687399 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.688694 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.691108 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.691693 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.691834 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:49.695885 (+ 4051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:49.695891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:49.695895 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:49.695928 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:49.700020 (+ 4092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:49.701287 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:49.701295 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:49.701299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:49.701476 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:49.702341 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:49.702350 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:49.702356 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:49.702876 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:49.703082 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:49.704053 (+ 971us) server_negotiation.cc:299] Negotiation successful
0927 21:26:49.704289 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 21:26:49.706840 2523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:49.680840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37339 (local address 127.0.97.3:48889)
0927 21:26:49.681779 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:49.681832 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:49.684222 (+ 2390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:49.685215 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:49.685224 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:49.686040 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:26:49.687107 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:49.687145 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.688979 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.688985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:49.690926 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:49.690940 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.692131 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.692137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:49.692366 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:49.693135 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:26:49.693165 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:26:49.695602 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:26:49.700218 (+ 4616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:49.700234 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:49.700254 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:49.701052 (+ 798us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:49.701775 (+ 723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:49.701780 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:49.701785 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:49.702168 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:49.703044 (+ 876us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:26:49.703051 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:26:49.703445 (+ 394us) client_negotiation.cc:769] Sending connection context
0927 21:26:49.703919 (+ 474us) client_negotiation.cc:240] Negotiation successful
0927 21:26:49.704618 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":431,"threads_started":1}
I20260927 21:26:49.732103 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:49.715249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51517 (local address 127.0.97.62:37339)
0927 21:26:49.715409 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:49.715414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:49.718253 (+ 2839us) server_negotiation.cc:407] Connection header received
0927 21:26:49.718494 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:49.718502 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:49.718604 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:49.718719 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:49.720440 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.721147 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.723388 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.724029 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.724167 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:49.726187 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:49.726195 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:49.726199 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:49.726239 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:49.729138 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:49.729983 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:49.729988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:49.729991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:49.730155 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:49.730676 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:49.730679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:49.730682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:49.730929 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:49.731090 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:49.731506 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 21:26:49.731724 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 21:26:49.732185 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:49.714718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37339 (local address 127.0.97.3:51517)
0927 21:26:49.718119 (+ 3401us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:49.718140 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:49.718325 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:49.718800 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:49.718808 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:49.719719 (+ 911us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:26:49.720230 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:49.720248 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.721306 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.721312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:49.723160 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:49.723183 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.724230 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.724238 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:49.724406 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:49.725097 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:26:49.725118 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:26:49.725988 (+ 870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:26:49.729274 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:49.729285 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:49.729289 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:49.729839 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:49.730297 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:49.730306 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:49.730313 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:49.730540 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:49.731056 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:26:49.731061 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:26:49.731237 (+ 176us) client_negotiation.cc:769] Sending connection context
0927 21:26:49.731388 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 21:26:49.731728 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3223,"thread_start_us":3090,"threads_started":1}
I20260927 21:26:49.738961 2499 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45051
I20260927 21:26:49.739035 2633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45051 every 8 connection(s)
I20260927 21:26:49.740209 2499 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:26:49.742326 2634 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37339
I20260927 21:26:49.742573 2634 heartbeater.cc:464] Registering TS with master...
I20260927 21:26:49.742719 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 2499
I20260927 21:26:49.742887 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-2/wal/instance
I20260927 21:26:49.743402 388 external_mini_cluster.cc:1428] Setting key 264a34917479da842232f54369006fd9
I20260927 21:26:49.743402 2634 heartbeater.cc:511] Master 127.0.97.62:37339 requested a full tablet report, sending...
I20260927 21:26:49.745115 2148 ts_manager.cc:194] Registered new tserver with Master: 71407dcfd18b4fad8d96b8b2b5ea45c8 (127.0.97.3:45051)
I20260927 21:26:49.746125 2148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48889
I20260927 21:26:49.746722 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 21:26:49 dist-test-slave-bxm0 krb5kdc[2104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:26:49 dist-test-slave-bxm0 krb5kdc[2104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:26:49.794231 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:49.780217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41032 (local address 127.0.97.62:37339)
0927 21:26:49.780386 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:49.780390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:49.780726 (+ 336us) server_negotiation.cc:407] Connection header received
0927 21:26:49.780853 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:49.780857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:49.780923 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:49.781021 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:49.782557 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.783396 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.784700 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.785094 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.785205 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:49.789253 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:49.789259 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:49.789262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:49.789299 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:49.791797 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:49.792500 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:49.792504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:49.792506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:49.792644 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:49.792975 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:49.792978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:49.792980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:49.793240 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:49.793419 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:49.793786 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 21:26:49.793951 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:26:49.798312 2148 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:26:49.805271 2148 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:26:49.841413 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:49.818538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49142 (local address 127.0.97.2:35833)
0927 21:26:49.834424 (+ 15886us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:49.834434 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:49.834462 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:26:49.834549 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:49.834560 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:49.834760 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:49.834939 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:26:49.835457 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.836660 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.838832 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.839975 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.840103 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:49.840312 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:49.840897 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 21:26:49.841112 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15713,"thread_start_us":15574,"threads_started":1}
I20260927 21:26:49.841491 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:49.818048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35833 (local address 127.0.0.1:49142)
0927 21:26:49.831370 (+ 13322us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:49.831394 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:49.831537 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:49.834926 (+ 3389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:49.834931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:49.834953 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:26:49.835296 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:49.835304 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.836904 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.836908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:49.838677 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:49.838690 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.840435 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.840443 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:49.840642 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:49.840697 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 21:26:49.840836 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 21:26:49.841032 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13153,"mutex_wait_us":3,"thread_start_us":12955,"threads_started":1}
I20260927 21:26:49.842044 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:49.817843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45051 (local address 127.0.0.1:59456)
0927 21:26:49.830205 (+ 12362us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:49.830278 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:49.830550 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:49.833417 (+ 2867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:49.833430 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:49.833487 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:26:49.834205 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:49.834217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.836380 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.836389 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:49.839332 (+ 2943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:49.839346 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.841170 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.841176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:49.841355 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:49.841388 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 21:26:49.841496 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 21:26:49.841655 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11982,"mutex_wait_us":164,"thread_start_us":12948,"threads_started":1}
I20260927 21:26:49.842800 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:49.818060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59456 (local address 127.0.97.3:45051)
0927 21:26:49.832134 (+ 14074us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:49.832148 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:49.832195 (+ 47us) server_negotiation.cc:407] Connection header received
0927 21:26:49.832329 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:49.832340 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:49.832715 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:49.832993 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:26:49.834446 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.836195 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.839518 (+ 3323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.840946 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.841077 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:49.841303 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:49.842305 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0927 21:26:49.842489 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13843,"thread_start_us":13643,"threads_started":1}
I20260927 21:26:49.844180 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:49.818842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46186 (local address 127.0.97.1:42437)
0927 21:26:49.832408 (+ 13566us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:49.832421 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:49.833190 (+ 769us) server_negotiation.cc:407] Connection header received
0927 21:26:49.833287 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:49.833294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:49.833524 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:49.833745 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:26:49.834648 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.836189 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.841396 (+ 5207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.843158 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.843284 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:49.843493 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:49.843706 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 21:26:49.843865 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13392,"thread_start_us":13273,"threads_started":1}
I20260927 21:26:49.845286 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:49.818268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42437 (local address 127.0.0.1:46186)
0927 21:26:49.832892 (+ 14624us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:49.832909 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:49.833065 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:49.833933 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:49.833936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:49.833966 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:26:49.834476 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:49.834491 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.836441 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.836448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:49.839332 (+ 2884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:49.839346 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.843348 (+ 4002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.843354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:49.843511 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:49.843543 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 21:26:49.843644 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 21:26:49.844922 (+ 1278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14507,"mutex_wait_us":226,"thread_start_us":14347,"threads_started":1}
I20260927 21:26:49.845665 2425 tablet_service.cc:1467] Processing CreateTablet for tablet 2909577c4df444e89de0a27ca3f9fb50 (DEFAULT_TABLE table=test-table [id=bf82013cdca24b928ae9d434cfc964ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:26:49.846397 2564 tablet_service.cc:1467] Processing CreateTablet for tablet 2909577c4df444e89de0a27ca3f9fb50 (DEFAULT_TABLE table=test-table [id=bf82013cdca24b928ae9d434cfc964ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:26:49.847055 2425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2909577c4df444e89de0a27ca3f9fb50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:49.847570 2564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2909577c4df444e89de0a27ca3f9fb50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:49.848708 2286 tablet_service.cc:1467] Processing CreateTablet for tablet 2909577c4df444e89de0a27ca3f9fb50 (DEFAULT_TABLE table=test-table [id=bf82013cdca24b928ae9d434cfc964ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:26:49.850028 2286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2909577c4df444e89de0a27ca3f9fb50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:49.858551 2653 tablet_bootstrap.cc:492] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8: Bootstrap starting.
I20260927 21:26:49.861922 2653 tablet_bootstrap.cc:654] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:49.863224 2653 log.cc:824] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8: Log is configured to *not* fsync() on all Append() calls
I20260927 21:26:49.865106 2656 tablet_bootstrap.cc:492] T 2909577c4df444e89de0a27ca3f9fb50 P 0a4fb22e29ef4818bf36fa4aca79c34e: Bootstrap starting.
I20260927 21:26:49.866751 2654 tablet_bootstrap.cc:492] T 2909577c4df444e89de0a27ca3f9fb50 P df4ce836ef04452a9334f9e3cb62ed5f: Bootstrap starting.
I20260927 21:26:49.869560 2656 tablet_bootstrap.cc:654] T 2909577c4df444e89de0a27ca3f9fb50 P 0a4fb22e29ef4818bf36fa4aca79c34e: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:49.870021 2654 tablet_bootstrap.cc:654] T 2909577c4df444e89de0a27ca3f9fb50 P df4ce836ef04452a9334f9e3cb62ed5f: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:49.870584 2653 tablet_bootstrap.cc:492] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8: No bootstrap required, opened a new log
I20260927 21:26:49.870867 2653 ts_tablet_manager.cc:1397] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20260927 21:26:49.871418 2654 log.cc:824] T 2909577c4df444e89de0a27ca3f9fb50 P df4ce836ef04452a9334f9e3cb62ed5f: Log is configured to *not* fsync() on all Append() calls
I20260927 21:26:49.871877 2656 log.cc:824] T 2909577c4df444e89de0a27ca3f9fb50 P 0a4fb22e29ef4818bf36fa4aca79c34e: Log is configured to *not* fsync() on all Append() calls
I20260927 21:26:49.874257 2654 tablet_bootstrap.cc:492] T 2909577c4df444e89de0a27ca3f9fb50 P df4ce836ef04452a9334f9e3cb62ed5f: No bootstrap required, opened a new log
I20260927 21:26:49.874536 2654 ts_tablet_manager.cc:1397] T 2909577c4df444e89de0a27ca3f9fb50 P df4ce836ef04452a9334f9e3cb62ed5f: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 21:26:49.882666 2654 raft_consensus.cc:348] T 2909577c4df444e89de0a27ca3f9fb50 P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } }
I20260927 21:26:49.883078 2654 raft_consensus.cc:374] T 2909577c4df444e89de0a27ca3f9fb50 P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:49.883172 2654 raft_consensus.cc:729] T 2909577c4df444e89de0a27ca3f9fb50 P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df4ce836ef04452a9334f9e3cb62ed5f, State: Initialized, Role: FOLLOWER
I20260927 21:26:49.883994 2654 consensus_queue.cc:260] T 2909577c4df444e89de0a27ca3f9fb50 P df4ce836ef04452a9334f9e3cb62ed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } }
I20260927 21:26:49.884639 2653 raft_consensus.cc:348] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } }
I20260927 21:26:49.885068 2653 raft_consensus.cc:374] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:49.885125 2495 heartbeater.cc:503] Master 127.0.97.62:37339 was elected leader, sending a full tablet report...
I20260927 21:26:49.885155 2653 raft_consensus.cc:729] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71407dcfd18b4fad8d96b8b2b5ea45c8, State: Initialized, Role: FOLLOWER
I20260927 21:26:49.885859 2654 ts_tablet_manager.cc:1428] T 2909577c4df444e89de0a27ca3f9fb50 P df4ce836ef04452a9334f9e3cb62ed5f: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260927 21:26:49.885880 2653 consensus_queue.cc:260] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } }
I20260927 21:26:49.886659 2656 tablet_bootstrap.cc:492] T 2909577c4df444e89de0a27ca3f9fb50 P 0a4fb22e29ef4818bf36fa4aca79c34e: No bootstrap required, opened a new log
I20260927 21:26:49.887034 2656 ts_tablet_manager.cc:1397] T 2909577c4df444e89de0a27ca3f9fb50 P 0a4fb22e29ef4818bf36fa4aca79c34e: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.007s
I20260927 21:26:49.894349 2634 heartbeater.cc:503] Master 127.0.97.62:37339 was elected leader, sending a full tablet report...
I20260927 21:26:49.895196 2653 ts_tablet_manager.cc:1428] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8: Time spent starting tablet: real 0.024s user 0.021s sys 0.003s
I20260927 21:26:49.898418 2656 raft_consensus.cc:348] T 2909577c4df444e89de0a27ca3f9fb50 P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } }
I20260927 21:26:49.898806 2656 raft_consensus.cc:374] T 2909577c4df444e89de0a27ca3f9fb50 P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:49.898922 2656 raft_consensus.cc:729] T 2909577c4df444e89de0a27ca3f9fb50 P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a4fb22e29ef4818bf36fa4aca79c34e, State: Initialized, Role: FOLLOWER
I20260927 21:26:49.899544 2656 consensus_queue.cc:260] T 2909577c4df444e89de0a27ca3f9fb50 P 0a4fb22e29ef4818bf36fa4aca79c34e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } }
I20260927 21:26:49.900970 2656 ts_tablet_manager.cc:1428] T 2909577c4df444e89de0a27ca3f9fb50 P 0a4fb22e29ef4818bf36fa4aca79c34e: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260927 21:26:49.940268 2357 tablet.cc:2369] T 2909577c4df444e89de0a27ca3f9fb50 P 0a4fb22e29ef4818bf36fa4aca79c34e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:26:49.957451 2496 tablet.cc:2369] T 2909577c4df444e89de0a27ca3f9fb50 P df4ce836ef04452a9334f9e3cb62ed5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:26:49.991892 2635 tablet.cc:2369] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:26:49.994278 2660 raft_consensus.cc:482] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:26:49.994630 2660 raft_consensus.cc:504] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } }
I20260927 21:26:49.996305 2660 leader_election.cc:288] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df4ce836ef04452a9334f9e3cb62ed5f (127.0.97.2:35833), 0a4fb22e29ef4818bf36fa4aca79c34e (127.0.97.1:42437)
I20260927 21:26:50.003295 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:49.996690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45823 (local address 127.0.97.2:35833)
0927 21:26:49.996895 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:49.996901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:49.997012 (+ 111us) server_negotiation.cc:407] Connection header received
0927 21:26:49.997215 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:49.997222 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:49.997323 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:49.997435 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:26:49.997927 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.998935 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.001306 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.002444 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.002587 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:50.002701 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:50.002890 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 21:26:50.003031 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 21:26:50.003293 2523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:49.996687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35833 (local address 127.0.97.3:45823)
0927 21:26:49.996896 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:49.996921 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:49.997040 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:49.997488 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:49.997492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:49.997537 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:26:49.997792 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:49.997807 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.999102 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.999106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:50.001090 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:50.001106 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.002605 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.002609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:50.002737 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:50.002762 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 21:26:50.002841 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 21:26:50.002938 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 21:26:50.003943 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:49.996693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56943 (local address 127.0.97.1:42437)
0927 21:26:49.996910 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:49.996918 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:49.997834 (+ 916us) server_negotiation.cc:407] Connection header received
0927 21:26:49.997982 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:49.997987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:49.998079 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:49.998183 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:26:49.998638 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.999462 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.001742 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.002980 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.003114 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:50.003211 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:50.003643 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 21:26:50.003783 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 21:26:50.004117 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:49.996689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42437 (local address 127.0.97.3:56943)
0927 21:26:49.997734 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:49.997755 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:49.997870 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:49.998198 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:49.998200 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:49.998218 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:26:49.998493 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:49.998500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:49.999603 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:49.999608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:50.001552 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:50.001563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.003210 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.003216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:50.003400 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:50.003438 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 21:26:50.003592 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 21:26:50.003731 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":225,"thread_start_us":385,"threads_started":1}
I20260927 21:26:50.004516 2450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2909577c4df444e89de0a27ca3f9fb50" candidate_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" is_pre_election: true
I20260927 21:26:50.005128 2450 raft_consensus.cc:2454] T 2909577c4df444e89de0a27ca3f9fb50 P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71407dcfd18b4fad8d96b8b2b5ea45c8 in term 0.
I20260927 21:26:50.005057 2311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2909577c4df444e89de0a27ca3f9fb50" candidate_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" is_pre_election: true
I20260927 21:26:50.005560 2311 raft_consensus.cc:2454] T 2909577c4df444e89de0a27ca3f9fb50 P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71407dcfd18b4fad8d96b8b2b5ea45c8 in term 0.
I20260927 21:26:50.005873 2518 leader_election.cc:302] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71407dcfd18b4fad8d96b8b2b5ea45c8, df4ce836ef04452a9334f9e3cb62ed5f; no voters:
I20260927 21:26:50.006258 2660 raft_consensus.cc:2781] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:26:50.006425 2660 raft_consensus.cc:482] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:26:50.006534 2660 raft_consensus.cc:3037] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:50.008772 2660 raft_consensus.cc:504] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } }
I20260927 21:26:50.009428 2660 leader_election.cc:288] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [CANDIDATE]: Term 1 election: Requested vote from peers df4ce836ef04452a9334f9e3cb62ed5f (127.0.97.2:35833), 0a4fb22e29ef4818bf36fa4aca79c34e (127.0.97.1:42437)
I20260927 21:26:50.009917 2450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2909577c4df444e89de0a27ca3f9fb50" candidate_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4ce836ef04452a9334f9e3cb62ed5f"
I20260927 21:26:50.010030 2311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2909577c4df444e89de0a27ca3f9fb50" candidate_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e"
I20260927 21:26:50.010152 2450 raft_consensus.cc:3037] T 2909577c4df444e89de0a27ca3f9fb50 P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:50.010296 2311 raft_consensus.cc:3037] T 2909577c4df444e89de0a27ca3f9fb50 P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:50.012342 2450 raft_consensus.cc:2454] T 2909577c4df444e89de0a27ca3f9fb50 P df4ce836ef04452a9334f9e3cb62ed5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71407dcfd18b4fad8d96b8b2b5ea45c8 in term 1.
I20260927 21:26:50.012523 2311 raft_consensus.cc:2454] T 2909577c4df444e89de0a27ca3f9fb50 P 0a4fb22e29ef4818bf36fa4aca79c34e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71407dcfd18b4fad8d96b8b2b5ea45c8 in term 1.
I20260927 21:26:50.012775 2518 leader_election.cc:302] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71407dcfd18b4fad8d96b8b2b5ea45c8, df4ce836ef04452a9334f9e3cb62ed5f; no voters:
I20260927 21:26:50.013083 2660 raft_consensus.cc:2781] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:26:50.013558 2660 raft_consensus.cc:686] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 1 LEADER]: Becoming Leader. State: Replica: 71407dcfd18b4fad8d96b8b2b5ea45c8, State: Running, Role: LEADER
I20260927 21:26:50.014205 2660 consensus_queue.cc:237] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } }
I20260927 21:26:50.019125 2148 catalog_manager.cc:5495] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71407dcfd18b4fad8d96b8b2b5ea45c8 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } health_report { overall_health: UNKNOWN } } }
Sep 27 21:26:50 dist-test-slave-bxm0 krb5kdc[2104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:26:50.090219 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:50.076140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41040 (local address 127.0.97.62:37339)
0927 21:26:50.076357 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:50.076362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:50.076984 (+ 622us) server_negotiation.cc:407] Connection header received
0927 21:26:50.077210 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:50.077215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:50.077295 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:50.077402 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:50.078689 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.079400 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.080628 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.081054 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.081192 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:50.084583 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:50.084588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:50.084592 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:50.084623 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:50.087461 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:50.088240 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:50.088244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:50.088246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:50.088367 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:50.088812 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:50.088816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:50.088818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:50.089102 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:50.089215 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:50.089777 (+ 562us) server_negotiation.cc:299] Negotiation successful
0927 21:26:50.089948 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 21:26:50.090222 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:50.076155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37339 (local address 127.0.0.1:41040)
0927 21:26:50.076858 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:50.076874 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:50.077068 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:50.077425 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:50.077429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:50.078278 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:26:50.078541 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:50.078551 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.079539 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.079544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:50.080485 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:50.080504 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.081234 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.081239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:50.081405 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:50.082151 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:26:50.082168 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:26:50.084373 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:26:50.087621 (+ 3248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:50.087633 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:50.087639 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:50.088110 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:50.088535 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:50.088539 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:50.088541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:50.088683 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:50.089248 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:26:50.089255 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:26:50.089408 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 21:26:50.089560 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 21:26:50.089853 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":422,"threads_started":1}
I20260927 21:26:50.095623 2148 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41040:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 21:26:50.096663 2148 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:26:50.103768 2286 tablet_service.cc:1467] Processing CreateTablet for tablet 3ea95347447b43508c989a030cafb2ae (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f00dd67a073e42eb931b4056835997c9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:26:50.103869 2425 tablet_service.cc:1467] Processing CreateTablet for tablet 3ea95347447b43508c989a030cafb2ae (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f00dd67a073e42eb931b4056835997c9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:26:50.104291 2564 tablet_service.cc:1467] Processing CreateTablet for tablet 3ea95347447b43508c989a030cafb2ae (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f00dd67a073e42eb931b4056835997c9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:26:50.104781 2425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ea95347447b43508c989a030cafb2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:50.104780 2286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ea95347447b43508c989a030cafb2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:50.105060 2564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ea95347447b43508c989a030cafb2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:50.110509 2656 tablet_bootstrap.cc:492] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e: Bootstrap starting.
I20260927 21:26:50.111966 2656 tablet_bootstrap.cc:654] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:50.112138 2654 tablet_bootstrap.cc:492] T 3ea95347447b43508c989a030cafb2ae P df4ce836ef04452a9334f9e3cb62ed5f: Bootstrap starting.
I20260927 21:26:50.112565 2653 tablet_bootstrap.cc:492] T 3ea95347447b43508c989a030cafb2ae P 71407dcfd18b4fad8d96b8b2b5ea45c8: Bootstrap starting.
I20260927 21:26:50.114140 2654 tablet_bootstrap.cc:654] T 3ea95347447b43508c989a030cafb2ae P df4ce836ef04452a9334f9e3cb62ed5f: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:50.114436 2653 tablet_bootstrap.cc:654] T 3ea95347447b43508c989a030cafb2ae P 71407dcfd18b4fad8d96b8b2b5ea45c8: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:50.115329 2656 tablet_bootstrap.cc:492] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e: No bootstrap required, opened a new log
I20260927 21:26:50.115518 2656 ts_tablet_manager.cc:1397] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260927 21:26:50.116490 2654 tablet_bootstrap.cc:492] T 3ea95347447b43508c989a030cafb2ae P df4ce836ef04452a9334f9e3cb62ed5f: No bootstrap required, opened a new log
I20260927 21:26:50.116644 2654 ts_tablet_manager.cc:1397] T 3ea95347447b43508c989a030cafb2ae P df4ce836ef04452a9334f9e3cb62ed5f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260927 21:26:50.116598 2656 raft_consensus.cc:348] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:50.116861 2656 raft_consensus.cc:374] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:50.116954 2656 raft_consensus.cc:729] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a4fb22e29ef4818bf36fa4aca79c34e, State: Initialized, Role: FOLLOWER
I20260927 21:26:50.117187 2656 consensus_queue.cc:260] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:50.117390 2654 raft_consensus.cc:348] T 3ea95347447b43508c989a030cafb2ae P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:50.117671 2654 raft_consensus.cc:374] T 3ea95347447b43508c989a030cafb2ae P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:50.117774 2654 raft_consensus.cc:729] T 3ea95347447b43508c989a030cafb2ae P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df4ce836ef04452a9334f9e3cb62ed5f, State: Initialized, Role: FOLLOWER
I20260927 21:26:50.117826 2661 tablet_replica.cc:439] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } } }
I20260927 21:26:50.117977 2654 consensus_queue.cc:260] T 3ea95347447b43508c989a030cafb2ae P df4ce836ef04452a9334f9e3cb62ed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:50.118208 2661 tablet_replica.cc:442] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:26:50.118593 2654 ts_tablet_manager.cc:1428] T 3ea95347447b43508c989a030cafb2ae P df4ce836ef04452a9334f9e3cb62ed5f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:26:50.118655 2659 tablet_replica.cc:439] T 3ea95347447b43508c989a030cafb2ae P df4ce836ef04452a9334f9e3cb62ed5f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } } }
I20260927 21:26:50.119024 2659 tablet_replica.cc:442] T 3ea95347447b43508c989a030cafb2ae P df4ce836ef04452a9334f9e3cb62ed5f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:26:50.119138 2656 ts_tablet_manager.cc:1428] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 21:26:50.121142 2653 tablet_bootstrap.cc:492] T 3ea95347447b43508c989a030cafb2ae P 71407dcfd18b4fad8d96b8b2b5ea45c8: No bootstrap required, opened a new log
I20260927 21:26:50.121299 2653 ts_tablet_manager.cc:1397] T 3ea95347447b43508c989a030cafb2ae P 71407dcfd18b4fad8d96b8b2b5ea45c8: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260927 21:26:50.122166 2653 raft_consensus.cc:348] T 3ea95347447b43508c989a030cafb2ae P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:50.122442 2653 raft_consensus.cc:374] T 3ea95347447b43508c989a030cafb2ae P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:50.122555 2653 raft_consensus.cc:729] T 3ea95347447b43508c989a030cafb2ae P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71407dcfd18b4fad8d96b8b2b5ea45c8, State: Initialized, Role: FOLLOWER
I20260927 21:26:50.122812 2653 consensus_queue.cc:260] T 3ea95347447b43508c989a030cafb2ae P 71407dcfd18b4fad8d96b8b2b5ea45c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:50.123212 2660 tablet_replica.cc:439] T 3ea95347447b43508c989a030cafb2ae P 71407dcfd18b4fad8d96b8b2b5ea45c8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } } }
I20260927 21:26:50.123417 2653 ts_tablet_manager.cc:1428] T 3ea95347447b43508c989a030cafb2ae P 71407dcfd18b4fad8d96b8b2b5ea45c8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:26:50.123510 2660 tablet_replica.cc:442] T 3ea95347447b43508c989a030cafb2ae P 71407dcfd18b4fad8d96b8b2b5ea45c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:26:50.182364 2661 raft_consensus.cc:482] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:26:50.182667 2661 raft_consensus.cc:504] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:50.184228 2661 leader_election.cc:288] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df4ce836ef04452a9334f9e3cb62ed5f (127.0.97.2:35833), 71407dcfd18b4fad8d96b8b2b5ea45c8 (127.0.97.3:45051)
I20260927 21:26:50.190565 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:50.184714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40659 (local address 127.0.97.3:45051)
0927 21:26:50.184890 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:50.184897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:50.185439 (+ 542us) server_negotiation.cc:407] Connection header received
0927 21:26:50.185730 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:50.185742 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:50.185804 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:50.185906 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:26:50.186432 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.187139 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.188870 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.189663 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.189812 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:50.189920 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:50.190181 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 21:26:50.190305 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:26:50.190680 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:50.184649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45051 (local address 127.0.97.1:40659)
0927 21:26:50.185307 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:50.185324 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:50.185528 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:50.185921 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:50.185925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:50.185950 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:26:50.186279 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:50.186287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.187265 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.187273 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:50.188715 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:50.188726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.189864 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.189870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:50.190028 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:50.190051 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 21:26:50.190155 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 21:26:50.190291 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":374,"threads_started":1}
I20260927 21:26:50.191051 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:50.184714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38091 (local address 127.0.97.2:35833)
0927 21:26:50.184890 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:50.184897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:50.186018 (+ 1121us) server_negotiation.cc:407] Connection header received
0927 21:26:50.186246 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:50.186250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:50.186318 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:50.186417 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:26:50.186905 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.187637 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.189282 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.190104 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.190252 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:50.190346 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:50.190643 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 21:26:50.190763 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:26:50.191118 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:50.184663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35833 (local address 127.0.97.1:38091)
0927 21:26:50.185867 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:50.185881 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:50.186098 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:50.186429 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:50.186433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:50.186466 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:26:50.186747 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:50.186754 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.187770 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.187776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:50.189125 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:50.189137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.190315 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.190321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:50.190488 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:50.190509 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 21:26:50.190609 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 21:26:50.190731 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1099,"mutex_wait_us":32,"thread_start_us":837,"threads_started":1}
I20260927 21:26:50.191517 2588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea95347447b43508c989a030cafb2ae" candidate_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" is_pre_election: true
I20260927 21:26:50.191749 2450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea95347447b43508c989a030cafb2ae" candidate_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" is_pre_election: true
I20260927 21:26:50.191983 2450 raft_consensus.cc:2454] T 3ea95347447b43508c989a030cafb2ae P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a4fb22e29ef4818bf36fa4aca79c34e in term 0.
I20260927 21:26:50.191956 2588 raft_consensus.cc:2454] T 3ea95347447b43508c989a030cafb2ae P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a4fb22e29ef4818bf36fa4aca79c34e in term 0.
I20260927 21:26:50.192495 2239 leader_election.cc:302] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0a4fb22e29ef4818bf36fa4aca79c34e, 71407dcfd18b4fad8d96b8b2b5ea45c8, df4ce836ef04452a9334f9e3cb62ed5f; no voters:
I20260927 21:26:50.192914 2661 raft_consensus.cc:2781] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:26:50.193059 2661 raft_consensus.cc:482] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:26:50.193178 2661 raft_consensus.cc:3037] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:50.195469 2661 raft_consensus.cc:504] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:50.196041 2661 leader_election.cc:288] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e [CANDIDATE]: Term 1 election: Requested vote from peers df4ce836ef04452a9334f9e3cb62ed5f (127.0.97.2:35833), 71407dcfd18b4fad8d96b8b2b5ea45c8 (127.0.97.3:45051)
I20260927 21:26:50.196556 2588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea95347447b43508c989a030cafb2ae" candidate_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8"
I20260927 21:26:50.196568 2450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ea95347447b43508c989a030cafb2ae" candidate_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4ce836ef04452a9334f9e3cb62ed5f"
I20260927 21:26:50.196755 2588 raft_consensus.cc:3037] T 3ea95347447b43508c989a030cafb2ae P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:50.196808 2450 raft_consensus.cc:3037] T 3ea95347447b43508c989a030cafb2ae P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:50.198921 2588 raft_consensus.cc:2454] T 3ea95347447b43508c989a030cafb2ae P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a4fb22e29ef4818bf36fa4aca79c34e in term 1.
I20260927 21:26:50.199041 2450 raft_consensus.cc:2454] T 3ea95347447b43508c989a030cafb2ae P df4ce836ef04452a9334f9e3cb62ed5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a4fb22e29ef4818bf36fa4aca79c34e in term 1.
I20260927 21:26:50.199380 2237 leader_election.cc:302] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a4fb22e29ef4818bf36fa4aca79c34e, 71407dcfd18b4fad8d96b8b2b5ea45c8; no voters:
I20260927 21:26:50.199681 2661 raft_consensus.cc:2781] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:26:50.200176 2661 raft_consensus.cc:686] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e [term 1 LEADER]: Becoming Leader. State: Replica: 0a4fb22e29ef4818bf36fa4aca79c34e, State: Running, Role: LEADER
I20260927 21:26:50.200685 2661 consensus_queue.cc:237] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:50.203995 2676 tablet_replica.cc:439] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e: TxnStatusTablet state changed. Reason: New leader 0a4fb22e29ef4818bf36fa4aca79c34e. Latest consensus state: current_term: 1 leader_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } } }
I20260927 21:26:50.204475 2676 tablet_replica.cc:442] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:26:50.205991 2147 catalog_manager.cc:5495] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a4fb22e29ef4818bf36fa4aca79c34e (127.0.97.1). New cstate: current_term: 1 leader_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } health_report { overall_health: UNKNOWN } } }
I20260927 21:26:50.210069 2678 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 21:26:50.210353 2678 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 21:26:50.227294 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:50.221523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42437 (local address 127.0.0.1:46198)
0927 21:26:50.221895 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:50.221912 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:50.222024 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:50.222614 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:50.222619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:50.222630 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:26:50.222806 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:50.222815 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.223826 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.223834 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:50.224816 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:50.224826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.225575 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.225581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:50.225778 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:50.225832 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:26:50.226730 (+ 898us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:26:50.226757 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 21:26:50.226857 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 21:26:50.227006 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241}
I20260927 21:26:50.227331 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:50.221580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46198 (local address 127.0.97.1:42437)
0927 21:26:50.222241 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:50.222245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:50.222259 (+ 14us) server_negotiation.cc:407] Connection header received
0927 21:26:50.222334 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:50.222338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:50.222444 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:50.222596 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:26:50.222948 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.223693 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.224966 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.225411 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.225541 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:50.226033 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:26:50.226627 (+ 594us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:26:50.226772 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:50.226888 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 21:26:50.226981 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558}
I20260927 21:26:50.235293 2450 raft_consensus.cc:1261] T 3ea95347447b43508c989a030cafb2ae P df4ce836ef04452a9334f9e3cb62ed5f [term 1 FOLLOWER]: Refusing update from remote peer 0a4fb22e29ef4818bf36fa4aca79c34e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:26:50.235311 2588 raft_consensus.cc:1261] T 3ea95347447b43508c989a030cafb2ae P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 1 FOLLOWER]: Refusing update from remote peer 0a4fb22e29ef4818bf36fa4aca79c34e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:26:50.236331 2676 consensus_queue.cc:1035] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:26:50.236711 2661 consensus_queue.cc:1035] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:26:50.248986 2660 tablet_replica.cc:439] T 3ea95347447b43508c989a030cafb2ae P 71407dcfd18b4fad8d96b8b2b5ea45c8: TxnStatusTablet state changed. Reason: New leader 0a4fb22e29ef4818bf36fa4aca79c34e. Latest consensus state: current_term: 1 leader_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } } }
I20260927 21:26:50.249297 2660 tablet_replica.cc:442] T 3ea95347447b43508c989a030cafb2ae P 71407dcfd18b4fad8d96b8b2b5ea45c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:26:50.251505 2659 tablet_replica.cc:439] T 3ea95347447b43508c989a030cafb2ae P df4ce836ef04452a9334f9e3cb62ed5f: TxnStatusTablet state changed. Reason: New leader 0a4fb22e29ef4818bf36fa4aca79c34e. Latest consensus state: current_term: 1 leader_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } } }
I20260927 21:26:50.251814 2659 tablet_replica.cc:442] T 3ea95347447b43508c989a030cafb2ae P df4ce836ef04452a9334f9e3cb62ed5f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:26:50.252017 2681 mvcc.cc:205] Tried to move back new op lower bound from 7334069904313741312 to 7334069904186613760. Current Snapshot: MvccSnapshot[applied={T|T < 7334069904313741312}]
I20260927 21:26:50.253113 2683 mvcc.cc:205] Tried to move back new op lower bound from 7334069904313741312 to 7334069904186613760. Current Snapshot: MvccSnapshot[applied={T|T < 7334069904313741312}]
I20260927 21:26:50.263311 2676 tablet_replica.cc:439] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } } }
I20260927 21:26:50.263752 2676 tablet_replica.cc:442] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:26:50.264947 2676 tablet_replica.cc:439] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } } }
I20260927 21:26:50.265205 2676 tablet_replica.cc:442] T 3ea95347447b43508c989a030cafb2ae P 0a4fb22e29ef4818bf36fa4aca79c34e: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:26:50.266304 2660 tablet_replica.cc:439] T 3ea95347447b43508c989a030cafb2ae P 71407dcfd18b4fad8d96b8b2b5ea45c8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } } }
I20260927 21:26:50.266594 2660 tablet_replica.cc:442] T 3ea95347447b43508c989a030cafb2ae P 71407dcfd18b4fad8d96b8b2b5ea45c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:26:50.273010 2659 tablet_replica.cc:439] T 3ea95347447b43508c989a030cafb2ae P df4ce836ef04452a9334f9e3cb62ed5f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } } }
I20260927 21:26:50.273295 2659 tablet_replica.cc:442] T 3ea95347447b43508c989a030cafb2ae P df4ce836ef04452a9334f9e3cb62ed5f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:26:50.282505 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:50.276338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59470 (local address 127.0.97.3:45051)
0927 21:26:50.276511 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:50.276520 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:50.276535 (+ 15us) server_negotiation.cc:407] Connection header received
0927 21:26:50.276656 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:50.276707 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:50.276814 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:50.276937 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:26:50.277356 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.278246 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.280178 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.280578 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.280681 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:50.281219 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:26:50.281873 (+ 654us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:26:50.281984 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:50.282122 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 21:26:50.282197 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 21:26:50.299894 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:50.294235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51209 (local address 127.0.97.1:42437)
0927 21:26:50.294402 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:50.294426 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:50.294946 (+ 520us) server_negotiation.cc:407] Connection header received
0927 21:26:50.295214 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:50.295218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:50.295283 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:50.295402 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:26:50.295900 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.296637 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.298062 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.298492 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.298642 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:50.299126 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:26:50.299234 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:26:50.299334 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:50.299490 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 21:26:50.299573 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:26:50.299954 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:50.294198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42437 (local address 127.0.97.3:51209)
0927 21:26:50.294833 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:50.294849 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:50.295036 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:50.295418 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:50.295422 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:50.295433 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:26:50.295718 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:50.295727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.296768 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.296776 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:50.297918 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:50.297930 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.298688 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.298697 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:50.298821 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:50.298877 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:26:50.299335 (+ 458us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:26:50.299355 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 21:26:50.299449 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 21:26:50.299585 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":335,"threads_started":1}
I20260927 21:26:50.310479 2450 raft_consensus.cc:1261] T 2909577c4df444e89de0a27ca3f9fb50 P df4ce836ef04452a9334f9e3cb62ed5f [term 1 FOLLOWER]: Refusing update from remote peer 71407dcfd18b4fad8d96b8b2b5ea45c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:26:50.310708 2311 raft_consensus.cc:1261] T 2909577c4df444e89de0a27ca3f9fb50 P 0a4fb22e29ef4818bf36fa4aca79c34e [term 1 FOLLOWER]: Refusing update from remote peer 71407dcfd18b4fad8d96b8b2b5ea45c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:26:50.311370 2660 consensus_queue.cc:1035] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:26:50.311678 2663 consensus_queue.cc:1035] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:26:50.362804 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:50.355538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34413 (local address 127.0.97.3:45051)
0927 21:26:50.355740 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:50.355748 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:50.356333 (+ 585us) server_negotiation.cc:407] Connection header received
0927 21:26:50.356701 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:50.356707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:50.356805 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:50.356898 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:26:50.357559 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.358680 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.360777 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.361496 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.361667 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:50.362099 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:26:50.362230 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:26:50.362322 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:50.362441 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 21:26:50.362521 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 21:26:50.362856 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:50.355436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45051 (local address 127.0.97.1:34413)
0927 21:26:50.356244 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:50.356263 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:50.356419 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:50.356919 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:50.356926 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:50.356940 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:26:50.357352 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:50.357365 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.358834 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.358839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:50.360530 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:50.360544 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.361727 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.361732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:50.361852 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:50.361909 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:26:50.362336 (+ 427us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:26:50.362349 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 21:26:50.362424 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 21:26:50.362541 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":438,"threads_started":1}
I20260927 21:26:50.400779 2148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:26:50.401086 2148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:26:50.405323 2148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3be07d0d94614d2891dc897eec726613: Sending DeleteTablet for 3 replicas of tablet 2909577c4df444e89de0a27ca3f9fb50
I20260927 21:26:50.406615 2564 tablet_service.cc:1514] Processing DeleteTablet for tablet 2909577c4df444e89de0a27ca3f9fb50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:26:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59456
I20260927 21:26:50.407660 2701 tablet_replica.cc:330] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8: stopping tablet replica
I20260927 21:26:50.407653 2425 tablet_service.cc:1514] Processing DeleteTablet for tablet 2909577c4df444e89de0a27ca3f9fb50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:26:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49142
I20260927 21:26:50.408265 2701 raft_consensus.cc:2229] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:26:50.408962 2286 tablet_service.cc:1514] Processing DeleteTablet for tablet 2909577c4df444e89de0a27ca3f9fb50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:26:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46186
I20260927 21:26:50.410349 2147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:26:50.411082 2147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:26:50.420020 2424 tablet_service.cc:1467] Processing CreateTablet for tablet 6621154bb94a48528e1036ce5e13fb69 (DEFAULT_TABLE table=test-table [id=2d09f9b11c4e4b57b77830007ab4331f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:26:50.419970 2285 tablet_service.cc:1467] Processing CreateTablet for tablet 6621154bb94a48528e1036ce5e13fb69 (DEFAULT_TABLE table=test-table [id=2d09f9b11c4e4b57b77830007ab4331f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:26:50.420528 2424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6621154bb94a48528e1036ce5e13fb69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:50.420554 2285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6621154bb94a48528e1036ce5e13fb69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:50.424718 2701 raft_consensus.cc:2258] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:26:50.426031 2564 tablet_service.cc:1467] Processing CreateTablet for tablet 6621154bb94a48528e1036ce5e13fb69 (DEFAULT_TABLE table=test-table [id=2d09f9b11c4e4b57b77830007ab4331f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:26:50.427043 2564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6621154bb94a48528e1036ce5e13fb69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:50.427983 2656 tablet_bootstrap.cc:492] T 6621154bb94a48528e1036ce5e13fb69 P 0a4fb22e29ef4818bf36fa4aca79c34e: Bootstrap starting.
I20260927 21:26:50.429819 2656 tablet_bootstrap.cc:654] T 6621154bb94a48528e1036ce5e13fb69 P 0a4fb22e29ef4818bf36fa4aca79c34e: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:50.431214 2701 ts_tablet_manager.cc:1905] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:26:50.434502 2653 tablet_bootstrap.cc:492] T 6621154bb94a48528e1036ce5e13fb69 P 71407dcfd18b4fad8d96b8b2b5ea45c8: Bootstrap starting.
I20260927 21:26:50.436091 2654 tablet_bootstrap.cc:492] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f: Bootstrap starting.
I20260927 21:26:50.436578 2653 tablet_bootstrap.cc:654] T 6621154bb94a48528e1036ce5e13fb69 P 71407dcfd18b4fad8d96b8b2b5ea45c8: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:50.438153 2654 tablet_bootstrap.cc:654] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:50.439308 2703 tablet_replica.cc:330] T 2909577c4df444e89de0a27ca3f9fb50 P 0a4fb22e29ef4818bf36fa4aca79c34e: stopping tablet replica
I20260927 21:26:50.443030 2703 raft_consensus.cc:2229] T 2909577c4df444e89de0a27ca3f9fb50 P 0a4fb22e29ef4818bf36fa4aca79c34e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:26:50.443444 2703 raft_consensus.cc:2258] T 2909577c4df444e89de0a27ca3f9fb50 P 0a4fb22e29ef4818bf36fa4aca79c34e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:26:50.445483 2654 tablet_bootstrap.cc:492] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f: No bootstrap required, opened a new log
I20260927 21:26:50.445744 2654 ts_tablet_manager.cc:1397] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260927 21:26:50.446561 2702 tablet_replica.cc:330] T 2909577c4df444e89de0a27ca3f9fb50 P df4ce836ef04452a9334f9e3cb62ed5f: stopping tablet replica
I20260927 21:26:50.446736 2654 raft_consensus.cc:348] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:50.446976 2654 raft_consensus.cc:374] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:50.447052 2654 raft_consensus.cc:729] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df4ce836ef04452a9334f9e3cb62ed5f, State: Initialized, Role: FOLLOWER
I20260927 21:26:50.447261 2702 raft_consensus.cc:2229] T 2909577c4df444e89de0a27ca3f9fb50 P df4ce836ef04452a9334f9e3cb62ed5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:26:50.447324 2654 consensus_queue.cc:260] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:50.447871 2654 ts_tablet_manager.cc:1428] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 21:26:50.448741 2703 ts_tablet_manager.cc:1905] T 2909577c4df444e89de0a27ca3f9fb50 P 0a4fb22e29ef4818bf36fa4aca79c34e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:26:50.450277 2656 tablet_bootstrap.cc:492] T 6621154bb94a48528e1036ce5e13fb69 P 0a4fb22e29ef4818bf36fa4aca79c34e: No bootstrap required, opened a new log
I20260927 21:26:50.450438 2656 ts_tablet_manager.cc:1397] T 6621154bb94a48528e1036ce5e13fb69 P 0a4fb22e29ef4818bf36fa4aca79c34e: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.007s
I20260927 21:26:50.451186 2656 raft_consensus.cc:348] T 6621154bb94a48528e1036ce5e13fb69 P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:50.451442 2656 raft_consensus.cc:374] T 6621154bb94a48528e1036ce5e13fb69 P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:50.451550 2656 raft_consensus.cc:729] T 6621154bb94a48528e1036ce5e13fb69 P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a4fb22e29ef4818bf36fa4aca79c34e, State: Initialized, Role: FOLLOWER
I20260927 21:26:50.451824 2656 consensus_queue.cc:260] T 6621154bb94a48528e1036ce5e13fb69 P 0a4fb22e29ef4818bf36fa4aca79c34e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:50.452503 2656 ts_tablet_manager.cc:1428] T 6621154bb94a48528e1036ce5e13fb69 P 0a4fb22e29ef4818bf36fa4aca79c34e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:26:50.456007 2653 tablet_bootstrap.cc:492] T 6621154bb94a48528e1036ce5e13fb69 P 71407dcfd18b4fad8d96b8b2b5ea45c8: No bootstrap required, opened a new log
I20260927 21:26:50.452042 2702 raft_consensus.cc:2258] T 2909577c4df444e89de0a27ca3f9fb50 P df4ce836ef04452a9334f9e3cb62ed5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:26:50.456211 2653 ts_tablet_manager.cc:1397] T 6621154bb94a48528e1036ce5e13fb69 P 71407dcfd18b4fad8d96b8b2b5ea45c8: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.005s
I20260927 21:26:50.457013 2653 raft_consensus.cc:348] T 6621154bb94a48528e1036ce5e13fb69 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:50.457283 2653 raft_consensus.cc:374] T 6621154bb94a48528e1036ce5e13fb69 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:50.457367 2653 raft_consensus.cc:729] T 6621154bb94a48528e1036ce5e13fb69 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71407dcfd18b4fad8d96b8b2b5ea45c8, State: Initialized, Role: FOLLOWER
I20260927 21:26:50.457695 2653 consensus_queue.cc:260] T 6621154bb94a48528e1036ce5e13fb69 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:50.458425 2702 ts_tablet_manager.cc:1905] T 2909577c4df444e89de0a27ca3f9fb50 P df4ce836ef04452a9334f9e3cb62ed5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:26:50.460363 2653 ts_tablet_manager.cc:1428] T 6621154bb94a48528e1036ce5e13fb69 P 71407dcfd18b4fad8d96b8b2b5ea45c8: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 21:26:50.462677 2701 ts_tablet_manager.cc:1918] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:26:50.462852 2701 log.cc:1196] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-2/wal/wals/2909577c4df444e89de0a27ca3f9fb50
I20260927 21:26:50.463429 2701 ts_tablet_manager.cc:1939] T 2909577c4df444e89de0a27ca3f9fb50 P 71407dcfd18b4fad8d96b8b2b5ea45c8: Deleting consensus metadata
I20260927 21:26:50.465111 2134 catalog_manager.cc:4841] TS 71407dcfd18b4fad8d96b8b2b5ea45c8 (127.0.97.3:45051): tablet 2909577c4df444e89de0a27ca3f9fb50 (table test-table [id=bf82013cdca24b928ae9d434cfc964ec]) successfully deleted
I20260927 21:26:50.465797 2703 ts_tablet_manager.cc:1918] T 2909577c4df444e89de0a27ca3f9fb50 P 0a4fb22e29ef4818bf36fa4aca79c34e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:26:50.466032 2703 log.cc:1196] T 2909577c4df444e89de0a27ca3f9fb50 P 0a4fb22e29ef4818bf36fa4aca79c34e: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-0/wal/wals/2909577c4df444e89de0a27ca3f9fb50
I20260927 21:26:50.466636 2702 ts_tablet_manager.cc:1918] T 2909577c4df444e89de0a27ca3f9fb50 P df4ce836ef04452a9334f9e3cb62ed5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:26:50.466792 2702 log.cc:1196] T 2909577c4df444e89de0a27ca3f9fb50 P df4ce836ef04452a9334f9e3cb62ed5f: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-1/wal/wals/2909577c4df444e89de0a27ca3f9fb50
I20260927 21:26:50.466959 2703 ts_tablet_manager.cc:1939] T 2909577c4df444e89de0a27ca3f9fb50 P 0a4fb22e29ef4818bf36fa4aca79c34e: Deleting consensus metadata
I20260927 21:26:50.467303 2702 ts_tablet_manager.cc:1939] T 2909577c4df444e89de0a27ca3f9fb50 P df4ce836ef04452a9334f9e3cb62ed5f: Deleting consensus metadata
I20260927 21:26:50.468894 2135 catalog_manager.cc:4841] TS 0a4fb22e29ef4818bf36fa4aca79c34e (127.0.97.1:42437): tablet 2909577c4df444e89de0a27ca3f9fb50 (table test-table [id=bf82013cdca24b928ae9d434cfc964ec]) successfully deleted
I20260927 21:26:50.468971 2136 catalog_manager.cc:4841] TS df4ce836ef04452a9334f9e3cb62ed5f (127.0.97.2:35833): tablet 2909577c4df444e89de0a27ca3f9fb50 (table test-table [id=bf82013cdca24b928ae9d434cfc964ec]) successfully deleted
I20260927 21:26:50.518337 2659 raft_consensus.cc:482] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:26:50.518643 2659 raft_consensus.cc:504] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:50.520522 2659 leader_election.cc:288] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a4fb22e29ef4818bf36fa4aca79c34e (127.0.97.1:42437), 71407dcfd18b4fad8d96b8b2b5ea45c8 (127.0.97.3:45051)
I20260927 21:26:50.541410 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:50.521289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44141 (local address 127.0.97.1:42437)
0927 21:26:50.521485 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:50.521491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:50.532056 (+ 10565us) server_negotiation.cc:407] Connection header received
0927 21:26:50.532548 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:50.532554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:50.532633 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:50.532726 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:26:50.533753 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.534949 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.538766 (+ 3817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.540148 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.540249 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:50.540357 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:50.540943 (+ 586us) server_negotiation.cc:299] Negotiation successful
0927 21:26:50.541089 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:26:50.541419 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:50.521407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43845 (local address 127.0.97.3:45051)
0927 21:26:50.521599 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:50.521606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:50.532416 (+ 10810us) server_negotiation.cc:407] Connection header received
0927 21:26:50.532738 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:50.532744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:50.532821 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:50.532905 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:26:50.533836 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.535006 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.538741 (+ 3735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.540142 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.540252 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:50.540384 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:50.541010 (+ 626us) server_negotiation.cc:299] Negotiation successful
0927 21:26:50.541144 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 21:26:50.542291 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:50.521022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45051 (local address 127.0.97.2:43845)
0927 21:26:50.532293 (+ 11271us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:50.532313 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:50.532548 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:50.533139 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:50.533143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:50.533170 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:26:50.533639 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:50.533651 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.535185 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.535194 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:50.538524 (+ 3330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:50.538541 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.540611 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.540617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:50.540811 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:50.540847 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 21:26:50.541701 (+ 854us) client_negotiation.cc:240] Negotiation successful
0927 21:26:50.541845 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11109,"mutex_wait_us":35,"thread_start_us":10593,"threads_started":1}
I20260927 21:26:50.542831 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:50.520874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42437 (local address 127.0.97.2:44141)
0927 21:26:50.531911 (+ 11037us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:50.531938 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:50.532353 (+ 415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:50.532932 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:50.532936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:50.532963 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:26:50.533514 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:50.533528 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.535134 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.535139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:50.538567 (+ 3428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:50.538582 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.540572 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.540579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:50.540749 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:50.540783 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 21:26:50.542087 (+ 1304us) client_negotiation.cc:240] Negotiation successful
0927 21:26:50.542239 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10877,"mutex_wait_us":285,"thread_start_us":11195,"threads_started":1}
I20260927 21:26:50.542985 2588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6621154bb94a48528e1036ce5e13fb69" candidate_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" is_pre_election: true
I20260927 21:26:50.543292 2588 raft_consensus.cc:2454] T 6621154bb94a48528e1036ce5e13fb69 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df4ce836ef04452a9334f9e3cb62ed5f in term 0.
I20260927 21:26:50.543648 2311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6621154bb94a48528e1036ce5e13fb69" candidate_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" is_pre_election: true
I20260927 21:26:50.543879 2311 raft_consensus.cc:2454] T 6621154bb94a48528e1036ce5e13fb69 P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df4ce836ef04452a9334f9e3cb62ed5f in term 0.
I20260927 21:26:50.543859 2377 leader_election.cc:302] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71407dcfd18b4fad8d96b8b2b5ea45c8, df4ce836ef04452a9334f9e3cb62ed5f; no voters:
I20260927 21:26:50.544291 2659 raft_consensus.cc:2781] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:26:50.544412 2659 raft_consensus.cc:482] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:26:50.544514 2659 raft_consensus.cc:3037] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:50.547757 2659 raft_consensus.cc:504] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:50.549402 2588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6621154bb94a48528e1036ce5e13fb69" candidate_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8"
I20260927 21:26:50.549667 2588 raft_consensus.cc:3037] T 6621154bb94a48528e1036ce5e13fb69 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:50.549885 2311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6621154bb94a48528e1036ce5e13fb69" candidate_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e"
I20260927 21:26:50.550093 2311 raft_consensus.cc:3037] T 6621154bb94a48528e1036ce5e13fb69 P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:50.552904 2588 raft_consensus.cc:2454] T 6621154bb94a48528e1036ce5e13fb69 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df4ce836ef04452a9334f9e3cb62ed5f in term 1.
I20260927 21:26:50.553332 2311 raft_consensus.cc:2454] T 6621154bb94a48528e1036ce5e13fb69 P 0a4fb22e29ef4818bf36fa4aca79c34e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df4ce836ef04452a9334f9e3cb62ed5f in term 1.
I20260927 21:26:50.553520 2377 leader_election.cc:302] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71407dcfd18b4fad8d96b8b2b5ea45c8, df4ce836ef04452a9334f9e3cb62ed5f; no voters:
I20260927 21:26:50.553980 2659 leader_election.cc:288] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f [CANDIDATE]: Term 1 election: Requested vote from peers 0a4fb22e29ef4818bf36fa4aca79c34e (127.0.97.1:42437), 71407dcfd18b4fad8d96b8b2b5ea45c8 (127.0.97.3:45051)
I20260927 21:26:50.554234 2659 raft_consensus.cc:2781] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:26:50.554486 2659 raft_consensus.cc:686] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f [term 1 LEADER]: Becoming Leader. State: Replica: df4ce836ef04452a9334f9e3cb62ed5f, State: Running, Role: LEADER
I20260927 21:26:50.554941 2659 consensus_queue.cc:237] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:50.562443 2147 catalog_manager.cc:5495] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f reported cstate change: term changed from 0 to 1, leader changed from <none> to df4ce836ef04452a9334f9e3cb62ed5f (127.0.97.2). New cstate: current_term: 1 leader_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } health_report { overall_health: UNKNOWN } } }
I20260927 21:26:50.604952 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:50.598012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49154 (local address 127.0.97.2:35833)
0927 21:26:50.598182 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:50.598188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:50.598208 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:26:50.598335 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:50.598340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:50.598442 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:50.598642 (+ 200us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:26:50.599055 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.600156 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.602186 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.602853 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.603005 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:50.603535 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:26:50.604277 (+ 742us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:26:50.604438 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:50.604560 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 21:26:50.604651 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 21:26:50.627255 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:50.617174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35407 (local address 127.0.97.1:42437)
0927 21:26:50.617393 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:50.617404 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:50.620759 (+ 3355us) server_negotiation.cc:407] Connection header received
0927 21:26:50.620992 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:50.620999 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:50.621086 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:50.621185 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:26:50.621724 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.622798 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.624811 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.625517 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.625675 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:50.626299 (+ 624us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:26:50.626474 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:26:50.626543 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:50.626815 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 21:26:50.626906 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 21:26:50.627414 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:50.617102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42437 (local address 127.0.97.2:35407)
0927 21:26:50.620624 (+ 3522us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:50.620649 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:50.620823 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:50.621227 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:50.621233 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:50.621246 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:26:50.621568 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:50.621581 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.622957 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.622962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:50.624636 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:50.624653 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.625765 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.625773 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:50.625969 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:50.626061 (+ 92us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:26:50.626658 (+ 597us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:26:50.626668 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 21:26:50.626742 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 21:26:50.626911 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3326,"thread_start_us":3192,"threads_started":1}
I20260927 21:26:50.638804 2588 raft_consensus.cc:1261] T 6621154bb94a48528e1036ce5e13fb69 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 1 FOLLOWER]: Refusing update from remote peer df4ce836ef04452a9334f9e3cb62ed5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:26:50.639312 2311 raft_consensus.cc:1261] T 6621154bb94a48528e1036ce5e13fb69 P 0a4fb22e29ef4818bf36fa4aca79c34e [term 1 FOLLOWER]: Refusing update from remote peer df4ce836ef04452a9334f9e3cb62ed5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:26:50.639636 2659 consensus_queue.cc:1035] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:26:50.643136 2659 consensus_queue.cc:1035] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:26:50.698388 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:50.684462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35833 (local address 127.0.97.1:56735)
0927 21:26:50.684658 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:50.684677 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:50.684792 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:50.685415 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:50.685421 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:50.685436 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:26:50.685680 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:50.685693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.690092 (+ 4399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.690100 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:50.691917 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:50.691934 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.696948 (+ 5014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.696957 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:50.697184 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:50.697239 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:26:50.697777 (+ 538us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:26:50.697790 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 21:26:50.697861 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 21:26:50.698038 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 21:26:50.699034 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:50.684804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56735 (local address 127.0.97.2:35833)
0927 21:26:50.685093 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:50.685099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:50.685119 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:26:50.685183 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:50.685191 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:50.685281 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:50.685393 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:26:50.685864 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.686928 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.692129 (+ 5201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:50.692903 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:50.693037 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:50.697441 (+ 4404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:26:50.697636 (+ 195us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:26:50.697723 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:50.698676 (+ 953us) server_negotiation.cc:299] Negotiation successful
0927 21:26:50.698764 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
W20260927 21:26:50.700161 2682 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 21:26:52.372049 2147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:26:52.372357 2147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:26:52.376997 2147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3be07d0d94614d2891dc897eec726613: Sending DeleteTablet for 3 replicas of tablet 6621154bb94a48528e1036ce5e13fb69
I20260927 21:26:52.377897 2425 tablet_service.cc:1514] Processing DeleteTablet for tablet 6621154bb94a48528e1036ce5e13fb69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:26:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49142
I20260927 21:26:52.377904 2286 tablet_service.cc:1514] Processing DeleteTablet for tablet 6621154bb94a48528e1036ce5e13fb69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:26:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46186
I20260927 21:26:52.378062 2564 tablet_service.cc:1514] Processing DeleteTablet for tablet 6621154bb94a48528e1036ce5e13fb69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:26:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59456
I20260927 21:26:52.378774 2742 tablet_replica.cc:330] T 6621154bb94a48528e1036ce5e13fb69 P 71407dcfd18b4fad8d96b8b2b5ea45c8: stopping tablet replica
I20260927 21:26:52.378904 2740 tablet_replica.cc:330] T 6621154bb94a48528e1036ce5e13fb69 P 0a4fb22e29ef4818bf36fa4aca79c34e: stopping tablet replica
I20260927 21:26:52.384716 2740 raft_consensus.cc:2229] T 6621154bb94a48528e1036ce5e13fb69 P 0a4fb22e29ef4818bf36fa4aca79c34e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:26:52.385066 2740 raft_consensus.cc:2258] T 6621154bb94a48528e1036ce5e13fb69 P 0a4fb22e29ef4818bf36fa4aca79c34e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:26:52.378933 2742 raft_consensus.cc:2229] T 6621154bb94a48528e1036ce5e13fb69 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:26:52.385418 2742 raft_consensus.cc:2258] T 6621154bb94a48528e1036ce5e13fb69 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:26:52.386452 2740 ts_tablet_manager.cc:1905] T 6621154bb94a48528e1036ce5e13fb69 P 0a4fb22e29ef4818bf36fa4aca79c34e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:26:52.386538 2742 ts_tablet_manager.cc:1905] T 6621154bb94a48528e1036ce5e13fb69 P 71407dcfd18b4fad8d96b8b2b5ea45c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:26:52.391765 2741 tablet_replica.cc:330] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f: stopping tablet replica
I20260927 21:26:52.392057 2741 raft_consensus.cc:2229] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:26:52.392706 2741 raft_consensus.cc:2258] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:26:52.392771 2742 ts_tablet_manager.cc:1918] T 6621154bb94a48528e1036ce5e13fb69 P 71407dcfd18b4fad8d96b8b2b5ea45c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:26:52.392901 2742 log.cc:1196] T 6621154bb94a48528e1036ce5e13fb69 P 71407dcfd18b4fad8d96b8b2b5ea45c8: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-2/wal/wals/6621154bb94a48528e1036ce5e13fb69
I20260927 21:26:52.393344 2742 ts_tablet_manager.cc:1939] T 6621154bb94a48528e1036ce5e13fb69 P 71407dcfd18b4fad8d96b8b2b5ea45c8: Deleting consensus metadata
I20260927 21:26:52.394050 2741 ts_tablet_manager.cc:1905] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:26:52.394577 2134 catalog_manager.cc:4841] TS 71407dcfd18b4fad8d96b8b2b5ea45c8 (127.0.97.3:45051): tablet 6621154bb94a48528e1036ce5e13fb69 (table test-table [id=2d09f9b11c4e4b57b77830007ab4331f]) successfully deleted
I20260927 21:26:52.394915 2740 ts_tablet_manager.cc:1918] T 6621154bb94a48528e1036ce5e13fb69 P 0a4fb22e29ef4818bf36fa4aca79c34e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:26:52.395063 2740 log.cc:1196] T 6621154bb94a48528e1036ce5e13fb69 P 0a4fb22e29ef4818bf36fa4aca79c34e: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-0/wal/wals/6621154bb94a48528e1036ce5e13fb69
I20260927 21:26:52.395498 2740 ts_tablet_manager.cc:1939] T 6621154bb94a48528e1036ce5e13fb69 P 0a4fb22e29ef4818bf36fa4aca79c34e: Deleting consensus metadata
I20260927 21:26:52.396732 2135 catalog_manager.cc:4841] TS 0a4fb22e29ef4818bf36fa4aca79c34e (127.0.97.1:42437): tablet 6621154bb94a48528e1036ce5e13fb69 (table test-table [id=2d09f9b11c4e4b57b77830007ab4331f]) successfully deleted
I20260927 21:26:52.399363 2741 ts_tablet_manager.cc:1918] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:26:52.399514 2741 log.cc:1196] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-1/wal/wals/6621154bb94a48528e1036ce5e13fb69
I20260927 21:26:52.399966 2741 ts_tablet_manager.cc:1939] T 6621154bb94a48528e1036ce5e13fb69 P df4ce836ef04452a9334f9e3cb62ed5f: Deleting consensus metadata
I20260927 21:26:52.401234 2136 catalog_manager.cc:4841] TS df4ce836ef04452a9334f9e3cb62ed5f (127.0.97.2:35833): tablet 6621154bb94a48528e1036ce5e13fb69 (table test-table [id=2d09f9b11c4e4b57b77830007ab4331f]) successfully deleted
Sep 27 21:26:52 dist-test-slave-bxm0 krb5kdc[2104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544412, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:26:52 dist-test-slave-bxm0 krb5kdc[2104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544412, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:26:52.435392 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:52.420718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60820 (local address 127.0.97.62:37339)
0927 21:26:52.421838 (+ 1120us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:52.421843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:52.421863 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:26:52.421915 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:52.421919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:52.421973 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:52.422100 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:52.423363 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:52.424140 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:52.425443 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:52.425930 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:52.426066 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:52.430111 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:52.430116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:52.430120 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:52.430157 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:52.432702 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:52.433415 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:52.433422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:52.433426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:52.433564 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:52.433995 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:52.434001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:52.434004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:52.434277 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:52.434522 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:52.434813 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 21:26:52.435036 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1013,"thread_start_us":760,"threads_started":1}
I20260927 21:26:52.439189 2147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:26:52.439811 2147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:26:52.447244 2286 tablet_service.cc:1467] Processing CreateTablet for tablet 4437c3e0326544a88327b9ff7ec1207d (DEFAULT_TABLE table=test-table [id=33825621c2da4bc58a95f0969b2e4632]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:26:52.447371 2425 tablet_service.cc:1467] Processing CreateTablet for tablet 4437c3e0326544a88327b9ff7ec1207d (DEFAULT_TABLE table=test-table [id=33825621c2da4bc58a95f0969b2e4632]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:26:52.447579 2564 tablet_service.cc:1467] Processing CreateTablet for tablet 4437c3e0326544a88327b9ff7ec1207d (DEFAULT_TABLE table=test-table [id=33825621c2da4bc58a95f0969b2e4632]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:26:52.447746 2286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4437c3e0326544a88327b9ff7ec1207d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:52.447801 2425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4437c3e0326544a88327b9ff7ec1207d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:52.447973 2564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4437c3e0326544a88327b9ff7ec1207d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:52.454046 2751 tablet_bootstrap.cc:492] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e: Bootstrap starting.
I20260927 21:26:52.455415 2751 tablet_bootstrap.cc:654] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:52.456108 2753 tablet_bootstrap.cc:492] T 4437c3e0326544a88327b9ff7ec1207d P df4ce836ef04452a9334f9e3cb62ed5f: Bootstrap starting.
I20260927 21:26:52.458223 2753 tablet_bootstrap.cc:654] T 4437c3e0326544a88327b9ff7ec1207d P df4ce836ef04452a9334f9e3cb62ed5f: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:52.458496 2752 tablet_bootstrap.cc:492] T 4437c3e0326544a88327b9ff7ec1207d P 71407dcfd18b4fad8d96b8b2b5ea45c8: Bootstrap starting.
I20260927 21:26:52.459097 2751 tablet_bootstrap.cc:492] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e: No bootstrap required, opened a new log
I20260927 21:26:52.459246 2751 ts_tablet_manager.cc:1397] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260927 21:26:52.460058 2752 tablet_bootstrap.cc:654] T 4437c3e0326544a88327b9ff7ec1207d P 71407dcfd18b4fad8d96b8b2b5ea45c8: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:52.459972 2751 raft_consensus.cc:348] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:52.460256 2751 raft_consensus.cc:374] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:52.460363 2751 raft_consensus.cc:729] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a4fb22e29ef4818bf36fa4aca79c34e, State: Initialized, Role: FOLLOWER
I20260927 21:26:52.460632 2753 tablet_bootstrap.cc:492] T 4437c3e0326544a88327b9ff7ec1207d P df4ce836ef04452a9334f9e3cb62ed5f: No bootstrap required, opened a new log
I20260927 21:26:52.460773 2753 ts_tablet_manager.cc:1397] T 4437c3e0326544a88327b9ff7ec1207d P df4ce836ef04452a9334f9e3cb62ed5f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 21:26:52.460657 2751 consensus_queue.cc:260] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:52.461248 2751 ts_tablet_manager.cc:1428] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260927 21:26:52.461489 2753 raft_consensus.cc:348] T 4437c3e0326544a88327b9ff7ec1207d P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:52.462181 2753 raft_consensus.cc:374] T 4437c3e0326544a88327b9ff7ec1207d P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:52.462298 2753 raft_consensus.cc:729] T 4437c3e0326544a88327b9ff7ec1207d P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df4ce836ef04452a9334f9e3cb62ed5f, State: Initialized, Role: FOLLOWER
I20260927 21:26:52.462540 2753 consensus_queue.cc:260] T 4437c3e0326544a88327b9ff7ec1207d P df4ce836ef04452a9334f9e3cb62ed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:52.463088 2753 ts_tablet_manager.cc:1428] T 4437c3e0326544a88327b9ff7ec1207d P df4ce836ef04452a9334f9e3cb62ed5f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:26:52.463438 2752 tablet_bootstrap.cc:492] T 4437c3e0326544a88327b9ff7ec1207d P 71407dcfd18b4fad8d96b8b2b5ea45c8: No bootstrap required, opened a new log
I20260927 21:26:52.463598 2752 ts_tablet_manager.cc:1397] T 4437c3e0326544a88327b9ff7ec1207d P 71407dcfd18b4fad8d96b8b2b5ea45c8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 21:26:52.464287 2752 raft_consensus.cc:348] T 4437c3e0326544a88327b9ff7ec1207d P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:52.464536 2752 raft_consensus.cc:374] T 4437c3e0326544a88327b9ff7ec1207d P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:52.464639 2752 raft_consensus.cc:729] T 4437c3e0326544a88327b9ff7ec1207d P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71407dcfd18b4fad8d96b8b2b5ea45c8, State: Initialized, Role: FOLLOWER
I20260927 21:26:52.464907 2752 consensus_queue.cc:260] T 4437c3e0326544a88327b9ff7ec1207d P 71407dcfd18b4fad8d96b8b2b5ea45c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:52.465857 2752 ts_tablet_manager.cc:1428] T 4437c3e0326544a88327b9ff7ec1207d P 71407dcfd18b4fad8d96b8b2b5ea45c8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:26:52.511313 2693 raft_consensus.cc:482] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:26:52.511565 2693 raft_consensus.cc:504] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:52.512324 2693 leader_election.cc:288] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df4ce836ef04452a9334f9e3cb62ed5f (127.0.97.2:35833), 71407dcfd18b4fad8d96b8b2b5ea45c8 (127.0.97.3:45051)
I20260927 21:26:52.512989 2588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4437c3e0326544a88327b9ff7ec1207d" candidate_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" is_pre_election: true
I20260927 21:26:52.512977 2450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4437c3e0326544a88327b9ff7ec1207d" candidate_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" is_pre_election: true
I20260927 21:26:52.513247 2588 raft_consensus.cc:2454] T 4437c3e0326544a88327b9ff7ec1207d P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a4fb22e29ef4818bf36fa4aca79c34e in term 0.
I20260927 21:26:52.513249 2450 raft_consensus.cc:2454] T 4437c3e0326544a88327b9ff7ec1207d P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a4fb22e29ef4818bf36fa4aca79c34e in term 0.
I20260927 21:26:52.513756 2239 leader_election.cc:302] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a4fb22e29ef4818bf36fa4aca79c34e, df4ce836ef04452a9334f9e3cb62ed5f; no voters:
I20260927 21:26:52.514037 2693 raft_consensus.cc:2781] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:26:52.514149 2693 raft_consensus.cc:482] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:26:52.514235 2693 raft_consensus.cc:3037] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:52.516168 2693 raft_consensus.cc:504] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:52.516690 2693 leader_election.cc:288] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e [CANDIDATE]: Term 1 election: Requested vote from peers df4ce836ef04452a9334f9e3cb62ed5f (127.0.97.2:35833), 71407dcfd18b4fad8d96b8b2b5ea45c8 (127.0.97.3:45051)
I20260927 21:26:52.517197 2450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4437c3e0326544a88327b9ff7ec1207d" candidate_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4ce836ef04452a9334f9e3cb62ed5f"
I20260927 21:26:52.517254 2588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4437c3e0326544a88327b9ff7ec1207d" candidate_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8"
I20260927 21:26:52.517437 2450 raft_consensus.cc:3037] T 4437c3e0326544a88327b9ff7ec1207d P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:52.517448 2588 raft_consensus.cc:3037] T 4437c3e0326544a88327b9ff7ec1207d P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:52.519732 2588 raft_consensus.cc:2454] T 4437c3e0326544a88327b9ff7ec1207d P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a4fb22e29ef4818bf36fa4aca79c34e in term 1.
I20260927 21:26:52.519778 2450 raft_consensus.cc:2454] T 4437c3e0326544a88327b9ff7ec1207d P df4ce836ef04452a9334f9e3cb62ed5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a4fb22e29ef4818bf36fa4aca79c34e in term 1.
I20260927 21:26:52.520274 2239 leader_election.cc:302] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a4fb22e29ef4818bf36fa4aca79c34e, df4ce836ef04452a9334f9e3cb62ed5f; no voters:
I20260927 21:26:52.520655 2693 raft_consensus.cc:2781] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:26:52.520809 2693 raft_consensus.cc:686] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e [term 1 LEADER]: Becoming Leader. State: Replica: 0a4fb22e29ef4818bf36fa4aca79c34e, State: Running, Role: LEADER
I20260927 21:26:52.521080 2693 consensus_queue.cc:237] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:52.523617 2147 catalog_manager.cc:5495] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a4fb22e29ef4818bf36fa4aca79c34e (127.0.97.1). New cstate: current_term: 1 leader_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } health_report { overall_health: UNKNOWN } } }
I20260927 21:26:52.547375 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:52.542411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41490 (local address 127.0.97.1:42437)
0927 21:26:52.543002 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:52.543006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:52.543023 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:26:52.543069 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:52.543072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:52.543128 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:52.543237 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:26:52.543635 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:52.544382 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:52.545663 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:52.546090 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:52.546230 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:52.546625 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:26:52.546735 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:26:52.546836 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:52.546997 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 21:26:52.547073 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":281,"threads_started":1}
I20260927 21:26:52.562249 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:52.556859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48883 (local address 127.0.97.1:42437)
0927 21:26:52.557090 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:52.557101 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:52.557381 (+ 280us) server_negotiation.cc:407] Connection header received
0927 21:26:52.557530 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:52.557533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:52.557598 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:52.557723 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:26:52.558169 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:52.558877 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:52.560690 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:52.561136 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:52.561230 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:52.561666 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:26:52.561774 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:26:52.561841 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:52.561961 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 21:26:52.562027 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":75}
I20260927 21:26:52.562341 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:52.556569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42437 (local address 127.0.97.1:48883)
0927 21:26:52.557289 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:52.557305 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:52.557420 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:52.557744 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:52.557748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:52.557758 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:26:52.558005 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:52.558013 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:52.559012 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:52.559017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:52.560562 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:52.560573 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:52.561315 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:52.561320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:52.561434 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:52.561482 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:26:52.561855 (+ 373us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:26:52.561863 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 21:26:52.561927 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 21:26:52.562034 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":429,"threads_started":1}
I20260927 21:26:52.570227 2588 raft_consensus.cc:1261] T 4437c3e0326544a88327b9ff7ec1207d P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 1 FOLLOWER]: Refusing update from remote peer 0a4fb22e29ef4818bf36fa4aca79c34e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:26:52.570223 2450 raft_consensus.cc:1261] T 4437c3e0326544a88327b9ff7ec1207d P df4ce836ef04452a9334f9e3cb62ed5f [term 1 FOLLOWER]: Refusing update from remote peer 0a4fb22e29ef4818bf36fa4aca79c34e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:26:52.570729 2693 consensus_queue.cc:1035] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:26:52.571039 2721 consensus_queue.cc:1035] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:26:52.617170 2149 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:26:52.617442 2149 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:26:52.621393 2149 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3be07d0d94614d2891dc897eec726613: Sending DeleteTablet for 3 replicas of tablet 4437c3e0326544a88327b9ff7ec1207d
I20260927 21:26:52.622416 2286 tablet_service.cc:1514] Processing DeleteTablet for tablet 4437c3e0326544a88327b9ff7ec1207d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:26:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46186
I20260927 21:26:52.622463 2425 tablet_service.cc:1514] Processing DeleteTablet for tablet 4437c3e0326544a88327b9ff7ec1207d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:26:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49142
I20260927 21:26:52.622745 2740 tablet_replica.cc:330] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e: stopping tablet replica
I20260927 21:26:52.622766 2741 tablet_replica.cc:330] T 4437c3e0326544a88327b9ff7ec1207d P df4ce836ef04452a9334f9e3cb62ed5f: stopping tablet replica
I20260927 21:26:52.622951 2740 raft_consensus.cc:2229] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:26:52.622958 2741 raft_consensus.cc:2229] T 4437c3e0326544a88327b9ff7ec1207d P df4ce836ef04452a9334f9e3cb62ed5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:26:52.623152 2741 raft_consensus.cc:2258] T 4437c3e0326544a88327b9ff7ec1207d P df4ce836ef04452a9334f9e3cb62ed5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:26:52.623488 2740 raft_consensus.cc:2258] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:26:52.624293 2741 ts_tablet_manager.cc:1905] T 4437c3e0326544a88327b9ff7ec1207d P df4ce836ef04452a9334f9e3cb62ed5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:26:52.624579 2740 ts_tablet_manager.cc:1905] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:26:52.624713 2564 tablet_service.cc:1514] Processing DeleteTablet for tablet 4437c3e0326544a88327b9ff7ec1207d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:26:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59456
I20260927 21:26:52.624989 2742 tablet_replica.cc:330] T 4437c3e0326544a88327b9ff7ec1207d P 71407dcfd18b4fad8d96b8b2b5ea45c8: stopping tablet replica
I20260927 21:26:52.625185 2742 raft_consensus.cc:2229] T 4437c3e0326544a88327b9ff7ec1207d P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:26:52.625357 2742 raft_consensus.cc:2258] T 4437c3e0326544a88327b9ff7ec1207d P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:26:52.625905 2149 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:26:52.626915 2149 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:26:52.631299 2741 ts_tablet_manager.cc:1918] T 4437c3e0326544a88327b9ff7ec1207d P df4ce836ef04452a9334f9e3cb62ed5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:26:52.631461 2741 log.cc:1196] T 4437c3e0326544a88327b9ff7ec1207d P df4ce836ef04452a9334f9e3cb62ed5f: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-1/wal/wals/4437c3e0326544a88327b9ff7ec1207d
I20260927 21:26:52.631880 2741 ts_tablet_manager.cc:1939] T 4437c3e0326544a88327b9ff7ec1207d P df4ce836ef04452a9334f9e3cb62ed5f: Deleting consensus metadata
I20260927 21:26:52.632158 2742 ts_tablet_manager.cc:1905] T 4437c3e0326544a88327b9ff7ec1207d P 71407dcfd18b4fad8d96b8b2b5ea45c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:26:52.633121 2136 catalog_manager.cc:4841] TS df4ce836ef04452a9334f9e3cb62ed5f (127.0.97.2:35833): tablet 4437c3e0326544a88327b9ff7ec1207d (table test-table [id=33825621c2da4bc58a95f0969b2e4632]) successfully deleted
I20260927 21:26:52.636708 2740 ts_tablet_manager.cc:1918] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:26:52.636890 2740 log.cc:1196] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-0/wal/wals/4437c3e0326544a88327b9ff7ec1207d
I20260927 21:26:52.637354 2740 ts_tablet_manager.cc:1939] T 4437c3e0326544a88327b9ff7ec1207d P 0a4fb22e29ef4818bf36fa4aca79c34e: Deleting consensus metadata
I20260927 21:26:52.638550 2135 catalog_manager.cc:4841] TS 0a4fb22e29ef4818bf36fa4aca79c34e (127.0.97.1:42437): tablet 4437c3e0326544a88327b9ff7ec1207d (table test-table [id=33825621c2da4bc58a95f0969b2e4632]) successfully deleted
I20260927 21:26:52.639559 2286 tablet_service.cc:1467] Processing CreateTablet for tablet 0a86f63fbe9749acbc59f6676167ffc7 (DEFAULT_TABLE table=test-table [id=8595623c234e45258a6260605f15243e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:26:52.639849 2425 tablet_service.cc:1467] Processing CreateTablet for tablet 0a86f63fbe9749acbc59f6676167ffc7 (DEFAULT_TABLE table=test-table [id=8595623c234e45258a6260605f15243e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:26:52.639973 2742 ts_tablet_manager.cc:1918] T 4437c3e0326544a88327b9ff7ec1207d P 71407dcfd18b4fad8d96b8b2b5ea45c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:26:52.640029 2286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a86f63fbe9749acbc59f6676167ffc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:52.640122 2742 log.cc:1196] T 4437c3e0326544a88327b9ff7ec1207d P 71407dcfd18b4fad8d96b8b2b5ea45c8: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-2/wal/wals/4437c3e0326544a88327b9ff7ec1207d
I20260927 21:26:52.640275 2425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a86f63fbe9749acbc59f6676167ffc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:52.640517 2742 ts_tablet_manager.cc:1939] T 4437c3e0326544a88327b9ff7ec1207d P 71407dcfd18b4fad8d96b8b2b5ea45c8: Deleting consensus metadata
I20260927 21:26:52.642678 2134 catalog_manager.cc:4841] TS 71407dcfd18b4fad8d96b8b2b5ea45c8 (127.0.97.3:45051): tablet 4437c3e0326544a88327b9ff7ec1207d (table test-table [id=33825621c2da4bc58a95f0969b2e4632]) successfully deleted
I20260927 21:26:52.643491 2564 tablet_service.cc:1467] Processing CreateTablet for tablet 0a86f63fbe9749acbc59f6676167ffc7 (DEFAULT_TABLE table=test-table [id=8595623c234e45258a6260605f15243e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:26:52.643926 2564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a86f63fbe9749acbc59f6676167ffc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:52.646976 2751 tablet_bootstrap.cc:492] T 0a86f63fbe9749acbc59f6676167ffc7 P 0a4fb22e29ef4818bf36fa4aca79c34e: Bootstrap starting.
I20260927 21:26:52.648442 2753 tablet_bootstrap.cc:492] T 0a86f63fbe9749acbc59f6676167ffc7 P df4ce836ef04452a9334f9e3cb62ed5f: Bootstrap starting.
I20260927 21:26:52.648950 2751 tablet_bootstrap.cc:654] T 0a86f63fbe9749acbc59f6676167ffc7 P 0a4fb22e29ef4818bf36fa4aca79c34e: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:52.650213 2753 tablet_bootstrap.cc:654] T 0a86f63fbe9749acbc59f6676167ffc7 P df4ce836ef04452a9334f9e3cb62ed5f: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:52.654393 2752 tablet_bootstrap.cc:492] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8: Bootstrap starting.
I20260927 21:26:52.656685 2752 tablet_bootstrap.cc:654] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:52.665126 2753 tablet_bootstrap.cc:492] T 0a86f63fbe9749acbc59f6676167ffc7 P df4ce836ef04452a9334f9e3cb62ed5f: No bootstrap required, opened a new log
I20260927 21:26:52.665393 2753 ts_tablet_manager.cc:1397] T 0a86f63fbe9749acbc59f6676167ffc7 P df4ce836ef04452a9334f9e3cb62ed5f: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.008s
I20260927 21:26:52.666493 2753 raft_consensus.cc:348] T 0a86f63fbe9749acbc59f6676167ffc7 P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:52.666852 2753 raft_consensus.cc:374] T 0a86f63fbe9749acbc59f6676167ffc7 P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:52.666957 2753 raft_consensus.cc:729] T 0a86f63fbe9749acbc59f6676167ffc7 P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df4ce836ef04452a9334f9e3cb62ed5f, State: Initialized, Role: FOLLOWER
I20260927 21:26:52.667325 2753 consensus_queue.cc:260] T 0a86f63fbe9749acbc59f6676167ffc7 P df4ce836ef04452a9334f9e3cb62ed5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:52.669538 2751 tablet_bootstrap.cc:492] T 0a86f63fbe9749acbc59f6676167ffc7 P 0a4fb22e29ef4818bf36fa4aca79c34e: No bootstrap required, opened a new log
I20260927 21:26:52.669857 2751 ts_tablet_manager.cc:1397] T 0a86f63fbe9749acbc59f6676167ffc7 P 0a4fb22e29ef4818bf36fa4aca79c34e: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.013s
I20260927 21:26:52.670799 2752 tablet_bootstrap.cc:492] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8: No bootstrap required, opened a new log
I20260927 21:26:52.670915 2753 ts_tablet_manager.cc:1428] T 0a86f63fbe9749acbc59f6676167ffc7 P df4ce836ef04452a9334f9e3cb62ed5f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 21:26:52.671025 2752 ts_tablet_manager.cc:1397] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.006s
I20260927 21:26:52.670923 2751 raft_consensus.cc:348] T 0a86f63fbe9749acbc59f6676167ffc7 P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:52.671345 2751 raft_consensus.cc:374] T 0a86f63fbe9749acbc59f6676167ffc7 P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:52.671475 2751 raft_consensus.cc:729] T 0a86f63fbe9749acbc59f6676167ffc7 P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a4fb22e29ef4818bf36fa4aca79c34e, State: Initialized, Role: FOLLOWER
I20260927 21:26:52.671816 2751 consensus_queue.cc:260] T 0a86f63fbe9749acbc59f6676167ffc7 P 0a4fb22e29ef4818bf36fa4aca79c34e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:52.672593 2751 ts_tablet_manager.cc:1428] T 0a86f63fbe9749acbc59f6676167ffc7 P 0a4fb22e29ef4818bf36fa4aca79c34e: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260927 21:26:52.671782 2752 raft_consensus.cc:348] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:52.673543 2752 raft_consensus.cc:374] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:52.674235 2752 raft_consensus.cc:729] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71407dcfd18b4fad8d96b8b2b5ea45c8, State: Initialized, Role: FOLLOWER
I20260927 21:26:52.674568 2752 consensus_queue.cc:260] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:52.675285 2752 ts_tablet_manager.cc:1428] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260927 21:26:52.693104 2757 raft_consensus.cc:482] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:26:52.693384 2757 raft_consensus.cc:504] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:52.694286 2757 leader_election.cc:288] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a4fb22e29ef4818bf36fa4aca79c34e (127.0.97.1:42437), df4ce836ef04452a9334f9e3cb62ed5f (127.0.97.2:35833)
I20260927 21:26:52.694772 2310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a86f63fbe9749acbc59f6676167ffc7" candidate_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" is_pre_election: true
I20260927 21:26:52.694962 2450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a86f63fbe9749acbc59f6676167ffc7" candidate_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" is_pre_election: true
I20260927 21:26:52.695101 2310 raft_consensus.cc:2454] T 0a86f63fbe9749acbc59f6676167ffc7 P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71407dcfd18b4fad8d96b8b2b5ea45c8 in term 0.
I20260927 21:26:52.695181 2450 raft_consensus.cc:2454] T 0a86f63fbe9749acbc59f6676167ffc7 P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71407dcfd18b4fad8d96b8b2b5ea45c8 in term 0.
I20260927 21:26:52.695662 2518 leader_election.cc:302] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71407dcfd18b4fad8d96b8b2b5ea45c8, df4ce836ef04452a9334f9e3cb62ed5f; no voters:
I20260927 21:26:52.696004 2757 raft_consensus.cc:2781] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:26:52.696127 2757 raft_consensus.cc:482] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:26:52.696244 2757 raft_consensus.cc:3037] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:52.699082 2757 raft_consensus.cc:504] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:52.699707 2757 leader_election.cc:288] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [CANDIDATE]: Term 1 election: Requested vote from peers 0a4fb22e29ef4818bf36fa4aca79c34e (127.0.97.1:42437), df4ce836ef04452a9334f9e3cb62ed5f (127.0.97.2:35833)
I20260927 21:26:52.700245 2450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a86f63fbe9749acbc59f6676167ffc7" candidate_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4ce836ef04452a9334f9e3cb62ed5f"
I20260927 21:26:52.700280 2310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a86f63fbe9749acbc59f6676167ffc7" candidate_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e"
I20260927 21:26:52.700470 2450 raft_consensus.cc:3037] T 0a86f63fbe9749acbc59f6676167ffc7 P df4ce836ef04452a9334f9e3cb62ed5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:52.700490 2310 raft_consensus.cc:3037] T 0a86f63fbe9749acbc59f6676167ffc7 P 0a4fb22e29ef4818bf36fa4aca79c34e [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:52.704015 2450 raft_consensus.cc:2454] T 0a86f63fbe9749acbc59f6676167ffc7 P df4ce836ef04452a9334f9e3cb62ed5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71407dcfd18b4fad8d96b8b2b5ea45c8 in term 1.
I20260927 21:26:52.704039 2310 raft_consensus.cc:2454] T 0a86f63fbe9749acbc59f6676167ffc7 P 0a4fb22e29ef4818bf36fa4aca79c34e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71407dcfd18b4fad8d96b8b2b5ea45c8 in term 1.
I20260927 21:26:52.704543 2518 leader_election.cc:302] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71407dcfd18b4fad8d96b8b2b5ea45c8, df4ce836ef04452a9334f9e3cb62ed5f; no voters:
I20260927 21:26:52.705264 2757 raft_consensus.cc:2781] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:26:52.706188 2757 raft_consensus.cc:686] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 1 LEADER]: Becoming Leader. State: Replica: 71407dcfd18b4fad8d96b8b2b5ea45c8, State: Running, Role: LEADER
I20260927 21:26:52.706733 2757 consensus_queue.cc:237] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } }
I20260927 21:26:52.709997 2148 catalog_manager.cc:5495] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71407dcfd18b4fad8d96b8b2b5ea45c8 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71407dcfd18b4fad8d96b8b2b5ea45c8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45051 } health_report { overall_health: HEALTHY } } }
I20260927 21:26:52.742170 2772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:52.735841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42966 (local address 127.0.97.3:45051)
0927 21:26:52.736607 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:52.736612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:52.736634 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:26:52.736691 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:52.736695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:52.736799 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:52.736920 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:26:52.737420 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:52.738360 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:52.739850 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:52.740342 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:52.740494 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:52.741250 (+ 756us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:26:52.741379 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:26:52.741495 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:52.741668 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 21:26:52.741750 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":456,"threads_started":1}
I20260927 21:26:52.757777 2450 raft_consensus.cc:1261] T 0a86f63fbe9749acbc59f6676167ffc7 P df4ce836ef04452a9334f9e3cb62ed5f [term 1 FOLLOWER]: Refusing update from remote peer 71407dcfd18b4fad8d96b8b2b5ea45c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:26:52.757782 2310 raft_consensus.cc:1261] T 0a86f63fbe9749acbc59f6676167ffc7 P 0a4fb22e29ef4818bf36fa4aca79c34e [term 1 FOLLOWER]: Refusing update from remote peer 71407dcfd18b4fad8d96b8b2b5ea45c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:26:52.758442 2757 consensus_queue.cc:1035] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df4ce836ef04452a9334f9e3cb62ed5f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:26:52.759032 2757 consensus_queue.cc:1035] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a4fb22e29ef4818bf36fa4aca79c34e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:26:52.781728 2723 mvcc.cc:205] Tried to move back new op lower bound from 7334069914649669632 to 7334069914449416192. Current Snapshot: MvccSnapshot[applied={T|T < 7334069914649669632}]
W20260927 21:26:52.800493 2722 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 21:26:54.387189 2148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:26:54.387406 2148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:26:54.391466 2148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3be07d0d94614d2891dc897eec726613: Sending DeleteTablet for 3 replicas of tablet 0a86f63fbe9749acbc59f6676167ffc7
I20260927 21:26:54.392352 2286 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a86f63fbe9749acbc59f6676167ffc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:26:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46186
I20260927 21:26:54.392517 2564 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a86f63fbe9749acbc59f6676167ffc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:26:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59456
I20260927 21:26:54.392575 2425 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a86f63fbe9749acbc59f6676167ffc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:26:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49142
I20260927 21:26:54.393168 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 2220
I20260927 21:26:54.405840 2799 tablet_replica.cc:330] T 0a86f63fbe9749acbc59f6676167ffc7 P 0a4fb22e29ef4818bf36fa4aca79c34e: stopping tablet replica
I20260927 21:26:54.406108 2799 raft_consensus.cc:2229] T 0a86f63fbe9749acbc59f6676167ffc7 P 0a4fb22e29ef4818bf36fa4aca79c34e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:26:54.405555 2801 tablet_replica.cc:330] T 0a86f63fbe9749acbc59f6676167ffc7 P df4ce836ef04452a9334f9e3cb62ed5f: stopping tablet replica
I20260927 21:26:54.406452 2801 raft_consensus.cc:2229] T 0a86f63fbe9749acbc59f6676167ffc7 P df4ce836ef04452a9334f9e3cb62ed5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:26:54.406533 2800 tablet_replica.cc:330] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8: stopping tablet replica
I20260927 21:26:54.406817 2800 raft_consensus.cc:2229] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:26:54.406865 2799 raft_consensus.cc:2258] T 0a86f63fbe9749acbc59f6676167ffc7 P 0a4fb22e29ef4818bf36fa4aca79c34e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:26:54.406898 2801 raft_consensus.cc:2258] T 0a86f63fbe9749acbc59f6676167ffc7 P df4ce836ef04452a9334f9e3cb62ed5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:26:54.407765 2800 raft_consensus.cc:2258] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:26:54.407951 2801 ts_tablet_manager.cc:1905] T 0a86f63fbe9749acbc59f6676167ffc7 P df4ce836ef04452a9334f9e3cb62ed5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:26:54.407954 2799 ts_tablet_manager.cc:1905] T 0a86f63fbe9749acbc59f6676167ffc7 P 0a4fb22e29ef4818bf36fa4aca79c34e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:26:54.408687 2800 ts_tablet_manager.cc:1905] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:26:54.414793 2801 ts_tablet_manager.cc:1918] T 0a86f63fbe9749acbc59f6676167ffc7 P df4ce836ef04452a9334f9e3cb62ed5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:26:54.414829 2799 ts_tablet_manager.cc:1918] T 0a86f63fbe9749acbc59f6676167ffc7 P 0a4fb22e29ef4818bf36fa4aca79c34e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:26:54.414958 2801 log.cc:1196] T 0a86f63fbe9749acbc59f6676167ffc7 P df4ce836ef04452a9334f9e3cb62ed5f: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-1/wal/wals/0a86f63fbe9749acbc59f6676167ffc7
I20260927 21:26:54.414963 2799 log.cc:1196] T 0a86f63fbe9749acbc59f6676167ffc7 P 0a4fb22e29ef4818bf36fa4aca79c34e: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-0/wal/wals/0a86f63fbe9749acbc59f6676167ffc7
I20260927 21:26:54.415290 2800 ts_tablet_manager.cc:1918] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:26:54.415446 2799 ts_tablet_manager.cc:1939] T 0a86f63fbe9749acbc59f6676167ffc7 P 0a4fb22e29ef4818bf36fa4aca79c34e: Deleting consensus metadata
I20260927 21:26:54.415467 2800 log.cc:1196] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790544390748963-388-0/minicluster-data/ts-2/wal/wals/0a86f63fbe9749acbc59f6676167ffc7
I20260927 21:26:54.415472 2801 ts_tablet_manager.cc:1939] T 0a86f63fbe9749acbc59f6676167ffc7 P df4ce836ef04452a9334f9e3cb62ed5f: Deleting consensus metadata
I20260927 21:26:54.416062 2800 ts_tablet_manager.cc:1939] T 0a86f63fbe9749acbc59f6676167ffc7 P 71407dcfd18b4fad8d96b8b2b5ea45c8: Deleting consensus metadata
I20260927 21:26:54.416550 2135 catalog_manager.cc:4841] TS 0a4fb22e29ef4818bf36fa4aca79c34e (127.0.97.1:42437): tablet 0a86f63fbe9749acbc59f6676167ffc7 (table test-table [id=8595623c234e45258a6260605f15243e]) successfully deleted
I20260927 21:26:54.416643 2136 catalog_manager.cc:4841] TS df4ce836ef04452a9334f9e3cb62ed5f (127.0.97.2:35833): tablet 0a86f63fbe9749acbc59f6676167ffc7 (table test-table [id=8595623c234e45258a6260605f15243e]) successfully deleted
I20260927 21:26:54.417075 2134 catalog_manager.cc:4841] TS 71407dcfd18b4fad8d96b8b2b5ea45c8 (127.0.97.3:45051): tablet 0a86f63fbe9749acbc59f6676167ffc7 (table test-table [id=8595623c234e45258a6260605f15243e]) successfully deleted
Sep 27 21:26:54 dist-test-slave-bxm0 krb5kdc[2104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544412, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 21:26:54.428237 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:54.396641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41500 (local address 127.0.97.1:42437)
0927 21:26:54.406079 (+ 9438us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:54.406085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:54.406121 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:26:54.406196 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:54.406204 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:54.406289 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:54.409760 (+ 3471us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:54.409980 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:54.411141 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:54.413108 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:54.413879 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:54.415256 (+ 1377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:54.422063 (+ 6807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:54.422081 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:54.422096 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:54.422139 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:54.424918 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:54.425648 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:54.425659 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:54.425662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:54.425798 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:54.426190 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:54.426200 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:54.426202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:54.426578 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:54.426854 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:54.427652 (+ 798us) server_negotiation.cc:299] Negotiation successful
0927 21:26:54.427883 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9301,"thread_start_us":9148,"threads_started":1}
W20260927 21:26:54.428970 2351 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41500
W20260927 21:26:54.429562 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 2220: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:26:54.429807 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 2220
I20260927 21:26:54.455557 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 2360
Sep 27 21:26:54 dist-test-slave-bxm0 krb5kdc[2104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544412, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 21:26:54.470853 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:54.456386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58000 (local address 127.0.97.2:35833)
0927 21:26:54.456972 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:54.456976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:54.456993 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:26:54.457047 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:54.457050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:54.457129 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:54.457244 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:54.458573 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:54.459279 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:54.460499 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:54.460868 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:54.460961 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:54.464955 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:54.464970 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:54.464986 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:54.465025 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:54.467963 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:54.468713 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:54.468722 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:54.468724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:54.468880 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:54.469259 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:54.469263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:54.469265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:54.469699 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:54.469972 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:54.470280 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 21:26:54.470479 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":327,"threads_started":1}
W20260927 21:26:54.471609 2490 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58000
W20260927 21:26:54.472352 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 2360: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:26:54.472502 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 2360
I20260927 21:26:54.495266 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 2499
Sep 27 21:26:54 dist-test-slave-bxm0 krb5kdc[2104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544412, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 21:26:54.511250 2805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:54.496088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42980 (local address 127.0.97.3:45051)
0927 21:26:54.496644 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:54.496648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:54.496664 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:26:54.496716 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:54.496720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:54.496782 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:54.496872 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:54.498268 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:54.499078 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:54.500411 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:54.500845 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:54.500955 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:54.505377 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:54.505393 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:54.505419 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:54.505463 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:54.508304 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:54.509174 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:54.509182 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:54.509185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:54.509312 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:54.509868 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:54.509871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:54.509873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:54.510241 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:54.510453 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:54.510817 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 21:26:54.510996 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":282,"threads_started":1}
W20260927 21:26:54.511940 2629 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42980
W20260927 21:26:54.512560 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 2499: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:26:54.512722 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 2499
I20260927 21:26:54.535132 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 2117
I20260927 21:26:54.536463 2189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:26:54.651836 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 2117
2026-09-27T21:26:54Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8358 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:26:54 dist-test-slave-bxm0 krb5kdc[2809](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:26:54 dist-test-slave-bxm0 krb5kdc[2809](info): set up 2 sockets
Sep 27 21:26:54 dist-test-slave-bxm0 krb5kdc[2809](info): commencing operation
krb5kdc: starting...
W20260927 21:26:55.750171 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.004s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:26:55 dist-test-slave-bxm0 krb5kdc[2809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:26:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:26:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:26:55.949482 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38597
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:26:56.280535 2822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:26:56.280860 2822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:26:56.281021 2822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:26:56.288548 2822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:26:56.288681 2822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:26:56.288764 2822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:26:56.288844 2822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:26:56.288901 2822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:26:56.299077 2822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:26:56.300565 2822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:26:56.301959 2822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:26:56.314132 2830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:56.314155 2828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:56.314200 2827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:26:56.319442 2822 server_base.cc:1034] running on GCE node
I20260927 21:26:56.320369 2822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:26:56.322088 2822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:26:56.323300 2822 hybrid_clock.cc:648] HybridClock initialized: now 1790544416323250 us; error 42 us; skew 500 ppm
I20260927 21:26:56.323628 2822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:26:56 dist-test-slave-bxm0 krb5kdc[2809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:26:56.326930 2822 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:26:56.328843 2822 webserver.cc:460] Webserver started at http://127.0.97.62:42777/ using document root <none> and password file <none>
I20260927 21:26:56.329672 2822 fs_manager.cc:359] Metadata directory not provided
I20260927 21:26:56.329816 2822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:26:56.330163 2822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:26:56.333319 2822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "06ef42021d7e49bfb912163c89c3e74b"
format_stamp: "Formatted at 2026-09-27 21:26:56 on dist-test-slave-bxm0"
server_key: "19a8d33173f9f6c07fd3168a0d8e21e1"
server_key_iv: "4778db545e25490c2e13e65bb8008cc5"
server_key_version: "encryptionkey@0"
I20260927 21:26:56.334244 2822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "06ef42021d7e49bfb912163c89c3e74b"
format_stamp: "Formatted at 2026-09-27 21:26:56 on dist-test-slave-bxm0"
server_key: "19a8d33173f9f6c07fd3168a0d8e21e1"
server_key_iv: "4778db545e25490c2e13e65bb8008cc5"
server_key_version: "encryptionkey@0"
I20260927 21:26:56.339746 2822 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:26:56.343492 2837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:56.344986 2822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:26:56.345192 2822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "06ef42021d7e49bfb912163c89c3e74b"
format_stamp: "Formatted at 2026-09-27 21:26:56 on dist-test-slave-bxm0"
server_key: "19a8d33173f9f6c07fd3168a0d8e21e1"
server_key_iv: "4778db545e25490c2e13e65bb8008cc5"
server_key_version: "encryptionkey@0"
I20260927 21:26:56.345386 2822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:26:56.367085 2822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:26:56.371644 2822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:26:56.371958 2822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:26:56.391342 2822 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38597
I20260927 21:26:56.391377 2888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38597 every 8 connection(s)
I20260927 21:26:56.392796 2822 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:26:56.396833 2889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:26:56.399103 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 2822
I20260927 21:26:56.399336 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:26:56.400089 388 external_mini_cluster.cc:1428] Setting key 3382f91b59d3dcea55f93ca027a40bcb
I20260927 21:26:56.411216 2889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06ef42021d7e49bfb912163c89c3e74b: Bootstrap starting.
I20260927 21:26:56.414925 2889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06ef42021d7e49bfb912163c89c3e74b: Neither blocks nor log segments found. Creating new log.
I20260927 21:26:56.416122 2889 log.cc:824] T 00000000000000000000000000000000 P 06ef42021d7e49bfb912163c89c3e74b: Log is configured to *not* fsync() on all Append() calls
Sep 27 21:26:56 dist-test-slave-bxm0 krb5kdc[2809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:26:56.419376 2889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06ef42021d7e49bfb912163c89c3e74b: No bootstrap required, opened a new log
I20260927 21:26:56.425698 2892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:56.402521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47818 (local address 127.0.97.62:38597)
0927 21:26:56.403439 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:56.403457 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:56.403513 (+ 56us) server_negotiation.cc:407] Connection header received
0927 21:26:56.405906 (+ 2393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:56.405938 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:56.406524 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:56.407020 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:56.408442 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:56.410480 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:56.411864 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:56.412930 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:56.413152 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:56.417689 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:56.417707 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:56.417726 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:56.417762 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:56.421127 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:56.422137 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:56.422145 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:56.422151 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:56.422331 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:56.422745 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:56.422749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:56.422750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:56.423310 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:56.423545 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:56.424038 (+ 493us) server_negotiation.cc:299] Negotiation successful
0927 21:26:56.424392 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":423,"threads_started":1}
I20260927 21:26:56.429093 2889 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06ef42021d7e49bfb912163c89c3e74b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ef42021d7e49bfb912163c89c3e74b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38597 } }
I20260927 21:26:56.429523 2889 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06ef42021d7e49bfb912163c89c3e74b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:26:56.429687 2889 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06ef42021d7e49bfb912163c89c3e74b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ef42021d7e49bfb912163c89c3e74b, State: Initialized, Role: FOLLOWER
I20260927 21:26:56.430248 2889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06ef42021d7e49bfb912163c89c3e74b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ef42021d7e49bfb912163c89c3e74b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38597 } }
I20260927 21:26:56.430486 2889 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06ef42021d7e49bfb912163c89c3e74b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:26:56.430627 2889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06ef42021d7e49bfb912163c89c3e74b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:26:56.430795 2889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06ef42021d7e49bfb912163c89c3e74b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:26:56.433015 2889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06ef42021d7e49bfb912163c89c3e74b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ef42021d7e49bfb912163c89c3e74b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38597 } }
I20260927 21:26:56.433578 2889 leader_election.cc:302] T 00000000000000000000000000000000 P 06ef42021d7e49bfb912163c89c3e74b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06ef42021d7e49bfb912163c89c3e74b; no voters:
I20260927 21:26:56.434207 2889 leader_election.cc:288] T 00000000000000000000000000000000 P 06ef42021d7e49bfb912163c89c3e74b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:26:56.434419 2894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06ef42021d7e49bfb912163c89c3e74b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:26:56.435706 2894 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06ef42021d7e49bfb912163c89c3e74b [term 1 LEADER]: Becoming Leader. State: Replica: 06ef42021d7e49bfb912163c89c3e74b, State: Running, Role: LEADER
I20260927 21:26:56.436304 2894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06ef42021d7e49bfb912163c89c3e74b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ef42021d7e49bfb912163c89c3e74b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38597 } }
I20260927 21:26:56.436578 2889 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06ef42021d7e49bfb912163c89c3e74b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:26:56.439040 2896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06ef42021d7e49bfb912163c89c3e74b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06ef42021d7e49bfb912163c89c3e74b. Latest consensus state: current_term: 1 leader_uuid: "06ef42021d7e49bfb912163c89c3e74b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ef42021d7e49bfb912163c89c3e74b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38597 } } }
I20260927 21:26:56.439425 2896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06ef42021d7e49bfb912163c89c3e74b [sys.catalog]: This master's current role is: LEADER
I20260927 21:26:56.439998 2895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06ef42021d7e49bfb912163c89c3e74b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06ef42021d7e49bfb912163c89c3e74b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ef42021d7e49bfb912163c89c3e74b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38597 } } }
I20260927 21:26:56.440325 2895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06ef42021d7e49bfb912163c89c3e74b [sys.catalog]: This master's current role is: LEADER
I20260927 21:26:56.440531 2901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:26:56.448904 2901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:26:56.461153 2901 catalog_manager.cc:1347] Generated new cluster ID: c1604d1401ad408cb66b2c9a231c3d0d
I20260927 21:26:56.461282 2901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:26:56.504303 2901 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:26:56.505774 2901 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:26:56.519111 2901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06ef42021d7e49bfb912163c89c3e74b: Generated new TSK 0
I20260927 21:26:56.520002 2901 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 21:26:56.570819 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38597
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:26:56.912331 2916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:26:56.912629 2916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:26:56.912811 2916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:26:56.920467 2916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:26:56.920585 2916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:26:56.920774 2916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:26:56.931381 2916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:26:56.932760 2916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:26:56.934208 2916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:26:56.948021 2924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:56.948108 2921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:56.948062 2922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:26:56.951344 2916 server_base.cc:1034] running on GCE node
I20260927 21:26:56.952090 2916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:26:56.953135 2916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:26:56.954385 2916 hybrid_clock.cc:648] HybridClock initialized: now 1790544416954330 us; error 51 us; skew 500 ppm
I20260927 21:26:56.954735 2916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:26:56 dist-test-slave-bxm0 krb5kdc[2809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:26:56.958053 2916 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 21:26:56.959764 2916 webserver.cc:460] Webserver started at http://127.0.97.1:45291/ using document root <none> and password file <none>
I20260927 21:26:56.960490 2916 fs_manager.cc:359] Metadata directory not provided
I20260927 21:26:56.960590 2916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:26:56.960853 2916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:26:56.963876 2916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/data/instance:
uuid: "e8f227f663064e49a47beeb65a5f886f"
format_stamp: "Formatted at 2026-09-27 21:26:56 on dist-test-slave-bxm0"
server_key: "b2a3fa5f86f4d43db1063f71da232500"
server_key_iv: "d0e00ee83e28a1be2dc614c79f664f78"
server_key_version: "encryptionkey@0"
I20260927 21:26:56.964661 2916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e8f227f663064e49a47beeb65a5f886f"
format_stamp: "Formatted at 2026-09-27 21:26:56 on dist-test-slave-bxm0"
server_key: "b2a3fa5f86f4d43db1063f71da232500"
server_key_iv: "d0e00ee83e28a1be2dc614c79f664f78"
server_key_version: "encryptionkey@0"
I20260927 21:26:56.970183 2916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 21:26:56.973925 2931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:56.975359 2916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:26:56.975589 2916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "e8f227f663064e49a47beeb65a5f886f"
format_stamp: "Formatted at 2026-09-27 21:26:56 on dist-test-slave-bxm0"
server_key: "b2a3fa5f86f4d43db1063f71da232500"
server_key_iv: "d0e00ee83e28a1be2dc614c79f664f78"
server_key_version: "encryptionkey@0"
I20260927 21:26:56.975811 2916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:26:57.007689 2916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:26:57.012303 2916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:26:57.012615 2916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:26:57.013698 2916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:26:57.015055 2916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:26:57.015126 2916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:57.015244 2916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:26:57.015311 2916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:57.050083 2916 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35211
I20260927 21:26:57.050144 3043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35211 every 8 connection(s)
I20260927 21:26:57.051415 2916 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:26:57.052448 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 2916
I20260927 21:26:57.052675 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/wal/instance
I20260927 21:26:57.053462 388 external_mini_cluster.cc:1428] Setting key 9889d075acdefe179b2c155bf0090f2a
Sep 27 21:26:57 dist-test-slave-bxm0 krb5kdc[2809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 21:26:57.074826 3046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:57.054769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38691 (local address 127.0.97.62:38597)
0927 21:26:57.055235 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:57.055241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:57.056175 (+ 934us) server_negotiation.cc:407] Connection header received
0927 21:26:57.058078 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:57.058084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:57.058166 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:57.058292 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:57.059798 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:57.060525 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:57.063292 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:57.063720 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:57.063842 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:57.068464 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:57.068470 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:57.068473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:57.068508 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:57.071186 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:57.072029 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:57.072034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:57.072037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:57.072152 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:57.072706 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:57.072710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:57.072712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:57.073012 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:57.073180 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:57.074220 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0927 21:26:57.074451 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":243,"threads_started":1}
I20260927 21:26:57.075927 3047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:57.055208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38597 (local address 127.0.97.1:38691)
0927 21:26:57.056010 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:57.056057 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:57.057808 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:57.058532 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:57.058541 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:57.059054 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:26:57.059625 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:57.059642 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:57.060701 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:57.060705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:57.063082 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:57.063101 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:57.063989 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:57.063998 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:57.064352 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:57.065184 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:26:57.065223 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:26:57.068178 (+ 2955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:26:57.071358 (+ 3180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:57.071368 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:57.071388 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:57.071873 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:57.072301 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:57.072308 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:57.072315 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:57.072543 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:57.073153 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:26:57.073161 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:26:57.073718 (+ 557us) client_negotiation.cc:769] Sending connection context
0927 21:26:57.074097 (+ 379us) client_negotiation.cc:240] Negotiation successful
0927 21:26:57.074656 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":350,"threads_started":1}
I20260927 21:26:57.077788 3044 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38597
I20260927 21:26:57.078125 3044 heartbeater.cc:464] Registering TS with master...
I20260927 21:26:57.078922 3044 heartbeater.cc:511] Master 127.0.97.62:38597 requested a full tablet report, sending...
I20260927 21:26:57.081362 2854 ts_manager.cc:194] Registered new tserver with Master: e8f227f663064e49a47beeb65a5f886f (127.0.97.1:35211)
I20260927 21:26:57.083603 2854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 21:26:57.100983 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38597
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:26:57.432367 3050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:26:57.432688 3050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:26:57.432792 3050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:26:57.440286 3050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:26:57.440407 3050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:26:57.440528 3050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:26:57.450740 3050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:26:57.451994 3050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:26:57.453259 3050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:26:57.466923 3055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:57.467028 3058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:57.466979 3056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:26:57.468142 3050 server_base.cc:1034] running on GCE node
I20260927 21:26:57.468791 3050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:26:57.469821 3050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:26:57.471060 3050 hybrid_clock.cc:648] HybridClock initialized: now 1790544417471012 us; error 47 us; skew 500 ppm
I20260927 21:26:57.471406 3050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:26:57 dist-test-slave-bxm0 krb5kdc[2809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:26:57.475085 3050 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 21:26:57.476745 3050 webserver.cc:460] Webserver started at http://127.0.97.2:44215/ using document root <none> and password file <none>
I20260927 21:26:57.477486 3050 fs_manager.cc:359] Metadata directory not provided
I20260927 21:26:57.477667 3050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:26:57.478008 3050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:26:57.481307 3050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/data/instance:
uuid: "a262f3f0b32a4099adc2d7dba083a07c"
format_stamp: "Formatted at 2026-09-27 21:26:57 on dist-test-slave-bxm0"
server_key: "9713d5eb3acb1d9cf3f592d18ea6e557"
server_key_iv: "24be1c2f4bd3f40d6bd9072bca7bf93c"
server_key_version: "encryptionkey@0"
I20260927 21:26:57.482213 3050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a262f3f0b32a4099adc2d7dba083a07c"
format_stamp: "Formatted at 2026-09-27 21:26:57 on dist-test-slave-bxm0"
server_key: "9713d5eb3acb1d9cf3f592d18ea6e557"
server_key_iv: "24be1c2f4bd3f40d6bd9072bca7bf93c"
server_key_version: "encryptionkey@0"
I20260927 21:26:57.487593 3050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:26:57.491359 3065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:57.492918 3050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:26:57.493130 3050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "a262f3f0b32a4099adc2d7dba083a07c"
format_stamp: "Formatted at 2026-09-27 21:26:57 on dist-test-slave-bxm0"
server_key: "9713d5eb3acb1d9cf3f592d18ea6e557"
server_key_iv: "24be1c2f4bd3f40d6bd9072bca7bf93c"
server_key_version: "encryptionkey@0"
I20260927 21:26:57.493369 3050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:26:57.509017 3050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:26:57.513365 3050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:26:57.513748 3050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:26:57.514741 3050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:26:57.516083 3050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:26:57.516186 3050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:57.516343 3050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:26:57.516424 3050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:57.550319 3050 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43929
I20260927 21:26:57.550354 3177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43929 every 8 connection(s)
I20260927 21:26:57.551658 3050 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:26:57.560418 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 3050
I20260927 21:26:57.560674 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/wal/instance
I20260927 21:26:57.561108 388 external_mini_cluster.cc:1428] Setting key bd39ffc110e137b6d9dfb8fba48ccf7d
Sep 27 21:26:57 dist-test-slave-bxm0 krb5kdc[2809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:26:57.572129 3046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:57.554254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47861 (local address 127.0.97.62:38597)
0927 21:26:57.554457 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:57.554463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:57.555664 (+ 1201us) server_negotiation.cc:407] Connection header received
0927 21:26:57.557392 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:57.557396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:57.557464 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:57.557553 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:57.559105 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:57.559785 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:57.561510 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:57.562052 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:57.562152 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:57.565547 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:57.565556 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:57.565571 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:57.565652 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:57.568732 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:57.569744 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:57.569749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:57.569751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:57.569879 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:57.570362 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:57.570366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:57.570368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:57.570631 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:57.570741 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:57.571687 (+ 946us) server_negotiation.cc:299] Negotiation successful
0927 21:26:57.571877 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 21:26:57.573796 3180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:57.554812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38597 (local address 127.0.97.2:47861)
0927 21:26:57.555496 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:57.555533 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:57.557128 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:57.557828 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:57.557837 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:57.558351 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:26:57.558915 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:57.558932 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:57.559924 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:57.559928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:57.561360 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:57.561372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:57.562266 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:57.562271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:57.562469 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:57.563140 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:26:57.563170 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:26:57.565319 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:26:57.568878 (+ 3559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:57.568891 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:57.568910 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:57.569573 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:57.569993 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:57.570000 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:57.570006 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:57.570246 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:57.570755 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:26:57.570766 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:26:57.571244 (+ 478us) client_negotiation.cc:769] Sending connection context
0927 21:26:57.571576 (+ 332us) client_negotiation.cc:240] Negotiation successful
0927 21:26:57.572183 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":276,"threads_started":1}
I20260927 21:26:57.575456 3178 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38597
I20260927 21:26:57.575829 3178 heartbeater.cc:464] Registering TS with master...
I20260927 21:26:57.576604 3178 heartbeater.cc:511] Master 127.0.97.62:38597 requested a full tablet report, sending...
I20260927 21:26:57.578137 2854 ts_manager.cc:194] Registered new tserver with Master: a262f3f0b32a4099adc2d7dba083a07c (127.0.97.2:43929)
I20260927 21:26:57.578970 2854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47861
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 21:26:57.604985 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38597
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:26:57.936167 3183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:26:57.936523 3183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:26:57.936686 3183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:26:57.944082 3183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:26:57.944285 3183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:26:57.944478 3183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:26:57.954633 3183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:26:57.955981 3183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:26:57.957278 3183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:26:57.970422 3188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:57.970448 3189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:57.970419 3191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:26:57.971678 3183 server_base.cc:1034] running on GCE node
I20260927 21:26:57.972290 3183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:26:57.973241 3183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:26:57.974507 3183 hybrid_clock.cc:648] HybridClock initialized: now 1790544417974429 us; error 66 us; skew 500 ppm
I20260927 21:26:57.974870 3183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:26:57 dist-test-slave-bxm0 krb5kdc[2809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:26:57.978302 3183 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 21:26:57.979811 3183 webserver.cc:460] Webserver started at http://127.0.97.3:43019/ using document root <none> and password file <none>
I20260927 21:26:57.980666 3183 fs_manager.cc:359] Metadata directory not provided
I20260927 21:26:57.980825 3183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:26:57.981194 3183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:26:57.984644 3183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/data/instance:
uuid: "7fc7d37492ce4fbbab755bfaa795da11"
format_stamp: "Formatted at 2026-09-27 21:26:57 on dist-test-slave-bxm0"
server_key: "fd430b5ab3ee5e37932a47b43b95b070"
server_key_iv: "fd34ee7dc80852cad701d73e8eecc5d5"
server_key_version: "encryptionkey@0"
I20260927 21:26:57.985682 3183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7fc7d37492ce4fbbab755bfaa795da11"
format_stamp: "Formatted at 2026-09-27 21:26:57 on dist-test-slave-bxm0"
server_key: "fd430b5ab3ee5e37932a47b43b95b070"
server_key_iv: "fd34ee7dc80852cad701d73e8eecc5d5"
server_key_version: "encryptionkey@0"
I20260927 21:26:57.991734 3183 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 21:26:57.995666 3198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:57.997188 3183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:26:57.997383 3183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "7fc7d37492ce4fbbab755bfaa795da11"
format_stamp: "Formatted at 2026-09-27 21:26:57 on dist-test-slave-bxm0"
server_key: "fd430b5ab3ee5e37932a47b43b95b070"
server_key_iv: "fd34ee7dc80852cad701d73e8eecc5d5"
server_key_version: "encryptionkey@0"
I20260927 21:26:57.997610 3183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:26:58.017649 3183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:26:58.022403 3183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:26:58.022722 3183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:26:58.023825 3183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:26:58.025346 3183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:26:58.025456 3183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:58.025632 3183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:26:58.025710 3183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 21:26:58.064085 3183 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44131
I20260927 21:26:58.064126 3310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44131 every 8 connection(s)
I20260927 21:26:58.065464 3183 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:26:58.074432 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 3183
I20260927 21:26:58.074671 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/wal/instance
I20260927 21:26:58.075078 388 external_mini_cluster.cc:1428] Setting key d769217099c4741db9006d9e11bf9a5a
Sep 27 21:26:58 dist-test-slave-bxm0 krb5kdc[2809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:26:58.087176 3044 heartbeater.cc:503] Master 127.0.97.62:38597 was elected leader, sending a full tablet report...
I20260927 21:26:58.090642 3046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:58.068206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52947 (local address 127.0.97.62:38597)
0927 21:26:58.068429 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:58.068434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:58.069665 (+ 1231us) server_negotiation.cc:407] Connection header received
0927 21:26:58.071572 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:58.071578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:58.071671 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:58.071783 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:58.073334 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:58.074165 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:58.076459 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:26:58.076852 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:26:58.077006 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:58.081291 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:26:58.081296 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:26:58.081300 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:26:58.081335 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:26:58.084847 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:58.086176 (+ 1329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:58.086187 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:58.086192 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:58.086464 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:26:58.087296 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:26:58.087302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:26:58.087308 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:26:58.087682 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:26:58.087819 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 21:26:58.089976 (+ 2157us) server_negotiation.cc:299] Negotiation successful
0927 21:26:58.090295 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 21:26:58.091377 3313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:26:58.068702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38597 (local address 127.0.97.3:52947)
0927 21:26:58.069449 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0927 21:26:58.069496 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:26:58.071272 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:26:58.072034 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:26:58.072043 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:26:58.072561 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:26:58.073160 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:58.073178 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:58.074324 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:58.074329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:58.076236 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:26:58.076250 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:26:58.077114 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:26:58.077122 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:26:58.077375 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:26:58.078261 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:26:58.078315 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:26:58.081046 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:26:58.085066 (+ 4020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:58.085079 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:58.085104 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:58.085980 (+ 876us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:58.086640 (+ 660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:26:58.086648 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:26:58.086655 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:26:58.086937 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:26:58.087893 (+ 956us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:26:58.087901 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:26:58.088663 (+ 762us) client_negotiation.cc:769] Sending connection context
0927 21:26:58.089004 (+ 341us) client_negotiation.cc:240] Negotiation successful
0927 21:26:58.089795 (+ 791us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":310,"threads_started":1}
I20260927 21:26:58.093123 3311 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38597
I20260927 21:26:58.093410 3311 heartbeater.cc:464] Registering TS with master...
I20260927 21:26:58.094189 3311 heartbeater.cc:511] Master 127.0.97.62:38597 requested a full tablet report, sending...
I20260927 21:26:58.095671 2854 ts_manager.cc:194] Registered new tserver with Master: 7fc7d37492ce4fbbab755bfaa795da11 (127.0.97.3:44131)
I20260927 21:26:58.096609 2854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52947
I20260927 21:26:58.103688 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 21:26:58.135154 3046 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:26:58.132671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47828 (local address 127.0.97.62:38597)
0927 21:26:58.132898 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:58.132903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:58.133352 (+ 449us) server_negotiation.cc:407] Connection header received
0927 21:26:58.133497 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:58.133501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:58.133579 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:58.133737 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:58.134955 (+ 1218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47828: BlockingRecv error: recv got EOF from 127.0.0.1:47828 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:26:58.138927 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 2916
W20260927 21:26:58.141883 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 2916: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:35211: server requires authentication, but client does not have Kerberos credentials available
I20260927 21:26:58.142124 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 2916
I20260927 21:26:58.157838 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 3050
W20260927 21:26:58.160452 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 3050: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:43929: server requires authentication, but client does not have Kerberos credentials available
I20260927 21:26:58.160653 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 3050
W20260927 21:26:58.160780 3323 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:26:58.158798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33638 (local address 127.0.97.2:43929)
0927 21:26:58.159338 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0927 21:26:58.159345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:26:58.159371 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:26:58.159432 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:26:58.159438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:26:58.159652 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:26:58.159789 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:26:58.160447 (+ 658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33638: BlockingRecv error: recv got EOF from 127.0.0.1:33638 (error 108)
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":284,"threads_started":1}
I20260927 21:26:58.176229 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 3183
W20260927 21:26:58.179301 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 3183: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:44131: server requires authentication, but client does not have Kerberos credentials available
I20260927 21:26:58.179461 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 3183
I20260927 21:26:58.195034 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 2822
I20260927 21:26:58.196363 2884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:26:58.297909 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 2822
2026-09-27T21:26:58Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3644 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:26:58 dist-test-slave-bxm0 krb5kdc[3328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:26:58 dist-test-slave-bxm0 krb5kdc[3328](info): set up 2 sockets
Sep 27 21:26:58 dist-test-slave-bxm0 krb5kdc[3328](info): commencing operation
krb5kdc: starting...
W20260927 21:26:59.402179 388 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:26:59 dist-test-slave-bxm0 krb5kdc[3328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:26:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:26:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:26:59.596632 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46835
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:26:59.923979 3341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:26:59.924348 3341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:26:59.924525 3341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:26:59.931979 3341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:26:59.932145 3341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:26:59.932255 3341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:26:59.932360 3341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:26:59.932452 3341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:26:59.942507 3341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:26:59.943945 3341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:26:59.945365 3341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:26:59.958230 3346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:59.958262 3349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:26:59.958307 3347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:26:59.959300 3341 server_base.cc:1034] running on GCE node
I20260927 21:26:59.960155 3341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:26:59.961654 3341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:26:59.962950 3341 hybrid_clock.cc:648] HybridClock initialized: now 1790544419962876 us; error 36 us; skew 500 ppm
I20260927 21:26:59.963335 3341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:26:59 dist-test-slave-bxm0 krb5kdc[3328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:26:59.971782 3341 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:26:59.973515 3341 webserver.cc:460] Webserver started at http://127.0.97.62:39461/ using document root <none> and password file <none>
I20260927 21:26:59.974303 3341 fs_manager.cc:359] Metadata directory not provided
I20260927 21:26:59.974457 3341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:26:59.974783 3341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:26:59.977985 3341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "816ec7b4f2c045569b0bc975cec851e5"
format_stamp: "Formatted at 2026-09-27 21:26:59 on dist-test-slave-bxm0"
server_key: "c5f96d728f5733a4fca4da6a950c940a"
server_key_iv: "4c2875c62788cfdaa75a655d754293ca"
server_key_version: "encryptionkey@0"
I20260927 21:26:59.978823 3341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "816ec7b4f2c045569b0bc975cec851e5"
format_stamp: "Formatted at 2026-09-27 21:26:59 on dist-test-slave-bxm0"
server_key: "c5f96d728f5733a4fca4da6a950c940a"
server_key_iv: "4c2875c62788cfdaa75a655d754293ca"
server_key_version: "encryptionkey@0"
I20260927 21:26:59.984242 3341 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:26:59.987916 3356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:26:59.989528 3341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:26:59.989773 3341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "816ec7b4f2c045569b0bc975cec851e5"
format_stamp: "Formatted at 2026-09-27 21:26:59 on dist-test-slave-bxm0"
server_key: "c5f96d728f5733a4fca4da6a950c940a"
server_key_iv: "4c2875c62788cfdaa75a655d754293ca"
server_key_version: "encryptionkey@0"
I20260927 21:26:59.990006 3341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:00.006671 3341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:00.011443 3341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:00.011747 3341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:00.030980 3341 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46835
I20260927 21:27:00.031026 3407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46835 every 8 connection(s)
I20260927 21:27:00.032430 3341 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:27:00.035531 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 3341
I20260927 21:27:00.035727 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:27:00.036506 388 external_mini_cluster.cc:1428] Setting key efd34758a57d198ed68ef040bf26be20
I20260927 21:27:00.036804 3408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:00.051883 3408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 816ec7b4f2c045569b0bc975cec851e5: Bootstrap starting.
Sep 27 21:27:00 dist-test-slave-bxm0 krb5kdc[3328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:27:00.056083 3408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 816ec7b4f2c045569b0bc975cec851e5: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:00.057461 3408 log.cc:824] T 00000000000000000000000000000000 P 816ec7b4f2c045569b0bc975cec851e5: Log is configured to *not* fsync() on all Append() calls
I20260927 21:27:00.061004 3408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 816ec7b4f2c045569b0bc975cec851e5: No bootstrap required, opened a new log
I20260927 21:27:00.062294 3411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:00.038558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38586 (local address 127.0.97.62:46835)
0927 21:27:00.039377 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:00.039390 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:00.039430 (+ 40us) server_negotiation.cc:407] Connection header received
0927 21:27:00.041751 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:00.041776 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:00.042251 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:00.042615 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:00.044451 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:00.045864 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:00.047628 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:00.048407 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:00.048562 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:00.053213 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:00.053224 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:00.053246 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:00.053293 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:00.057409 (+ 4116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:00.058401 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:00.058422 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:00.058429 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:00.058614 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:00.059061 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:00.059066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:00.059069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:00.059774 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:00.060065 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:00.060476 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 21:27:00.060891 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":385,"threads_started":1}
I20260927 21:27:00.066453 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:27:00.069468 3408 raft_consensus.cc:348] T 00000000000000000000000000000000 P 816ec7b4f2c045569b0bc975cec851e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816ec7b4f2c045569b0bc975cec851e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46835 } }
I20260927 21:27:00.069914 3408 raft_consensus.cc:374] T 00000000000000000000000000000000 P 816ec7b4f2c045569b0bc975cec851e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:00.070020 3408 raft_consensus.cc:729] T 00000000000000000000000000000000 P 816ec7b4f2c045569b0bc975cec851e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816ec7b4f2c045569b0bc975cec851e5, State: Initialized, Role: FOLLOWER
I20260927 21:27:00.070627 3408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 816ec7b4f2c045569b0bc975cec851e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816ec7b4f2c045569b0bc975cec851e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46835 } }
I20260927 21:27:00.070849 3408 raft_consensus.cc:388] T 00000000000000000000000000000000 P 816ec7b4f2c045569b0bc975cec851e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:27:00.070983 3408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 816ec7b4f2c045569b0bc975cec851e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:27:00.071161 3408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 816ec7b4f2c045569b0bc975cec851e5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:00.073385 3408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 816ec7b4f2c045569b0bc975cec851e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816ec7b4f2c045569b0bc975cec851e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46835 } }
I20260927 21:27:00.074025 3408 leader_election.cc:302] T 00000000000000000000000000000000 P 816ec7b4f2c045569b0bc975cec851e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 816ec7b4f2c045569b0bc975cec851e5; no voters:
I20260927 21:27:00.074687 3408 leader_election.cc:288] T 00000000000000000000000000000000 P 816ec7b4f2c045569b0bc975cec851e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:27:00.074808 3414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 816ec7b4f2c045569b0bc975cec851e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:27:00.076061 3414 raft_consensus.cc:686] T 00000000000000000000000000000000 P 816ec7b4f2c045569b0bc975cec851e5 [term 1 LEADER]: Becoming Leader. State: Replica: 816ec7b4f2c045569b0bc975cec851e5, State: Running, Role: LEADER
I20260927 21:27:00.076682 3414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 816ec7b4f2c045569b0bc975cec851e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816ec7b4f2c045569b0bc975cec851e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46835 } }
I20260927 21:27:00.077015 3408 sys_catalog.cc:564] T 00000000000000000000000000000000 P 816ec7b4f2c045569b0bc975cec851e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:27:00.079427 3416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 816ec7b4f2c045569b0bc975cec851e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 816ec7b4f2c045569b0bc975cec851e5. Latest consensus state: current_term: 1 leader_uuid: "816ec7b4f2c045569b0bc975cec851e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816ec7b4f2c045569b0bc975cec851e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46835 } } }
I20260927 21:27:00.079870 3416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 816ec7b4f2c045569b0bc975cec851e5 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:00.080441 3415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 816ec7b4f2c045569b0bc975cec851e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "816ec7b4f2c045569b0bc975cec851e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816ec7b4f2c045569b0bc975cec851e5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46835 } } }
I20260927 21:27:00.080749 3415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 816ec7b4f2c045569b0bc975cec851e5 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:00.086253 3419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:27:00.093676 3419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:27:00.105968 3419 catalog_manager.cc:1347] Generated new cluster ID: e4d711cf2cb640d7b0df8972d2cbe0e4
I20260927 21:27:00.106102 3419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:27:00.121382 3411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:00.112024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38600 (local address 127.0.97.62:46835)
0927 21:27:00.112209 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:00.112220 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:00.112847 (+ 627us) server_negotiation.cc:407] Connection header received
0927 21:27:00.113036 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:00.113041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:00.113137 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:00.113249 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:00.114486 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:00.115610 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:00.117529 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:00.118179 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:00.118306 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:00.119596 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:00.119604 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:00.119608 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:27:00.120510 (+ 902us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:00.120568 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:27:00.120606 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:00.120714 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:00.120882 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 21:27:00.120990 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":66}
I20260927 21:27:00.138478 3419 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:27:00.140460 3419 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:27:00.154211 3419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 816ec7b4f2c045569b0bc975cec851e5: Generated new TSK 0
I20260927 21:27:00.155366 3419 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:27:00.166038 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 3341
I20260927 21:27:00.167327 3403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:00.259078 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 3341
2026-09-27T21:27:00Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1954 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:27:00 dist-test-slave-bxm0 krb5kdc[3443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:27:00 dist-test-slave-bxm0 krb5kdc[3443](info): set up 2 sockets
Sep 27 21:27:00 dist-test-slave-bxm0 krb5kdc[3443](info): commencing operation
krb5kdc: starting...
W20260927 21:27:01.338047 388 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:27:01 dist-test-slave-bxm0 krb5kdc[3443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:27:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:27:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:27:01.539544 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33257
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:01.869529 3456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:01.869865 3456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:01.870011 3456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:01.877327 3456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:27:01.877447 3456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:01.877528 3456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:01.877643 3456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:27:01.877712 3456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:27:01.887633 3456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:01.889031 3456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:01.890375 3456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:01.902230 3461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:01.902395 3462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:01.902418 3464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:01.903479 3456 server_base.cc:1034] running on GCE node
I20260927 21:27:01.904421 3456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:01.906157 3456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:01.907426 3456 hybrid_clock.cc:648] HybridClock initialized: now 1790544421907354 us; error 63 us; skew 500 ppm
I20260927 21:27:01.907817 3456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:01 dist-test-slave-bxm0 krb5kdc[3443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:01.911104 3456 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:27:01.912676 3456 webserver.cc:460] Webserver started at http://127.0.97.62:38827/ using document root <none> and password file <none>
I20260927 21:27:01.913352 3456 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:01.913452 3456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:01.913794 3456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:01.916692 3456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "ecf6be2c8a9041448dff45fcedd2b6c6"
format_stamp: "Formatted at 2026-09-27 21:27:01 on dist-test-slave-bxm0"
server_key: "f438b04ea47cb313c54771a71d70c41a"
server_key_iv: "fe07e2ef10d918e09e89ef8d333e72b7"
server_key_version: "encryptionkey@0"
I20260927 21:27:01.917377 3456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "ecf6be2c8a9041448dff45fcedd2b6c6"
format_stamp: "Formatted at 2026-09-27 21:27:01 on dist-test-slave-bxm0"
server_key: "f438b04ea47cb313c54771a71d70c41a"
server_key_iv: "fe07e2ef10d918e09e89ef8d333e72b7"
server_key_version: "encryptionkey@0"
I20260927 21:27:01.922513 3456 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260927 21:27:01.926262 3471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:01.927685 3456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:27:01.927896 3456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "ecf6be2c8a9041448dff45fcedd2b6c6"
format_stamp: "Formatted at 2026-09-27 21:27:01 on dist-test-slave-bxm0"
server_key: "f438b04ea47cb313c54771a71d70c41a"
server_key_iv: "fe07e2ef10d918e09e89ef8d333e72b7"
server_key_version: "encryptionkey@0"
I20260927 21:27:01.928090 3456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:01.966396 3456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:01.970911 3456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:01.971230 3456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:01.990327 3456 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33257
I20260927 21:27:01.990367 3522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33257 every 8 connection(s)
I20260927 21:27:01.991770 3456 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:27:01.995790 3523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:01.999715 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 3456
I20260927 21:27:01.999969 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:27:02.000869 388 external_mini_cluster.cc:1428] Setting key de129a648e569939ef6d5b8d375aee30
I20260927 21:27:02.009794 3523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecf6be2c8a9041448dff45fcedd2b6c6: Bootstrap starting.
I20260927 21:27:02.013159 3523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecf6be2c8a9041448dff45fcedd2b6c6: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:02.014387 3523 log.cc:824] T 00000000000000000000000000000000 P ecf6be2c8a9041448dff45fcedd2b6c6: Log is configured to *not* fsync() on all Append() calls
Sep 27 21:27:02 dist-test-slave-bxm0 krb5kdc[3443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:27:02.018010 3523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecf6be2c8a9041448dff45fcedd2b6c6: No bootstrap required, opened a new log
I20260927 21:27:02.024463 3526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:02.003335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41830 (local address 127.0.97.62:33257)
0927 21:27:02.004172 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:02.004186 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:02.004230 (+ 44us) server_negotiation.cc:407] Connection header received
0927 21:27:02.006101 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:02.006126 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:02.006581 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:02.007027 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:02.008452 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:02.009848 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:02.011163 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:02.011886 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:02.012104 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:02.016661 (+ 4557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:02.016669 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:02.016686 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:02.016725 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:02.020120 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:02.021109 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:02.021116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:02.021122 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:02.021284 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:02.021736 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:02.021739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:02.021741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:02.022264 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:02.022513 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:02.022975 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 21:27:02.023302 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":366,"threads_started":1}
I20260927 21:27:02.027666 3523 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecf6be2c8a9041448dff45fcedd2b6c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf6be2c8a9041448dff45fcedd2b6c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33257 } }
I20260927 21:27:02.028080 3523 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecf6be2c8a9041448dff45fcedd2b6c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:02.028200 3523 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecf6be2c8a9041448dff45fcedd2b6c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf6be2c8a9041448dff45fcedd2b6c6, State: Initialized, Role: FOLLOWER
I20260927 21:27:02.028821 3523 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecf6be2c8a9041448dff45fcedd2b6c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf6be2c8a9041448dff45fcedd2b6c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33257 } }
I20260927 21:27:02.029086 3523 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecf6be2c8a9041448dff45fcedd2b6c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:27:02.029241 3523 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecf6be2c8a9041448dff45fcedd2b6c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:27:02.029423 3523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecf6be2c8a9041448dff45fcedd2b6c6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:02.032251 3523 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecf6be2c8a9041448dff45fcedd2b6c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf6be2c8a9041448dff45fcedd2b6c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33257 } }
I20260927 21:27:02.032866 3523 leader_election.cc:302] T 00000000000000000000000000000000 P ecf6be2c8a9041448dff45fcedd2b6c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecf6be2c8a9041448dff45fcedd2b6c6; no voters:
I20260927 21:27:02.033595 3523 leader_election.cc:288] T 00000000000000000000000000000000 P ecf6be2c8a9041448dff45fcedd2b6c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:27:02.033759 3528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecf6be2c8a9041448dff45fcedd2b6c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:27:02.035133 3528 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecf6be2c8a9041448dff45fcedd2b6c6 [term 1 LEADER]: Becoming Leader. State: Replica: ecf6be2c8a9041448dff45fcedd2b6c6, State: Running, Role: LEADER
I20260927 21:27:02.035645 3528 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecf6be2c8a9041448dff45fcedd2b6c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf6be2c8a9041448dff45fcedd2b6c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33257 } }
I20260927 21:27:02.036123 3523 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecf6be2c8a9041448dff45fcedd2b6c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:27:02.038264 3530 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecf6be2c8a9041448dff45fcedd2b6c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecf6be2c8a9041448dff45fcedd2b6c6. Latest consensus state: current_term: 1 leader_uuid: "ecf6be2c8a9041448dff45fcedd2b6c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf6be2c8a9041448dff45fcedd2b6c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33257 } } }
I20260927 21:27:02.038666 3530 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecf6be2c8a9041448dff45fcedd2b6c6 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:02.039155 3529 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecf6be2c8a9041448dff45fcedd2b6c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecf6be2c8a9041448dff45fcedd2b6c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf6be2c8a9041448dff45fcedd2b6c6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33257 } } }
I20260927 21:27:02.039491 3529 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecf6be2c8a9041448dff45fcedd2b6c6 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:02.039572 3535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:27:02.045926 3535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:27:02.057456 3535 catalog_manager.cc:1347] Generated new cluster ID: 70a1e3d49bdd4ed69e474b5a56fa4936
I20260927 21:27:02.057596 3535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:27:02.078152 3535 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:27:02.079546 3535 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:27:02.090653 3535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecf6be2c8a9041448dff45fcedd2b6c6: Generated new TSK 0
I20260927 21:27:02.091536 3535 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 21:27:02.145581 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33257
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:02.504904 3549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:02.505208 3549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:02.505349 3549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:02.512826 3549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:02.512959 3549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:02.513155 3549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:27:02.523267 3549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:02.524663 3549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:02.526077 3549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:02.539407 3555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:02.539474 3557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:02.539515 3554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:02.540396 3549 server_base.cc:1034] running on GCE node
I20260927 21:27:02.540969 3549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:02.542053 3549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:02.543272 3549 hybrid_clock.cc:648] HybridClock initialized: now 1790544422543203 us; error 63 us; skew 500 ppm
I20260927 21:27:02.543642 3549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:02 dist-test-slave-bxm0 krb5kdc[3443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:02.547097 3549 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 21:27:02.548677 3549 webserver.cc:460] Webserver started at http://127.0.97.1:42987/ using document root <none> and password file <none>
I20260927 21:27:02.549486 3549 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:02.549685 3549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:02.550004 3549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:02.552995 3549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/data/instance:
uuid: "9c53b7f796a1448eaa9d661415b59df5"
format_stamp: "Formatted at 2026-09-27 21:27:02 on dist-test-slave-bxm0"
server_key: "10244bc96de80db2d91a6a1166810bc9"
server_key_iv: "3a410673195f5d1eec1c8866b78161b5"
server_key_version: "encryptionkey@0"
I20260927 21:27:02.553918 3549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9c53b7f796a1448eaa9d661415b59df5"
format_stamp: "Formatted at 2026-09-27 21:27:02 on dist-test-slave-bxm0"
server_key: "10244bc96de80db2d91a6a1166810bc9"
server_key_iv: "3a410673195f5d1eec1c8866b78161b5"
server_key_version: "encryptionkey@0"
I20260927 21:27:02.559216 3549 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 21:27:02.563122 3564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:02.564687 3549 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:27:02.564882 3549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "9c53b7f796a1448eaa9d661415b59df5"
format_stamp: "Formatted at 2026-09-27 21:27:02 on dist-test-slave-bxm0"
server_key: "10244bc96de80db2d91a6a1166810bc9"
server_key_iv: "3a410673195f5d1eec1c8866b78161b5"
server_key_version: "encryptionkey@0"
I20260927 21:27:02.565112 3549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:02.585985 3549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:02.590838 3549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:02.591176 3549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:02.592182 3549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:27:02.593575 3549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:02.593828 3549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 21:27:02.593976 3549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:02.594062 3549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:02.632308 3549 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38379
I20260927 21:27:02.632583 3676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38379 every 8 connection(s)
I20260927 21:27:02.634445 3549 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:27:02.641271 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 3549
I20260927 21:27:02.641505 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-0/wal/instance
I20260927 21:27:02.642392 388 external_mini_cluster.cc:1428] Setting key 3a0e61e347c22798f330403b4cab21e3
Sep 27 21:27:02 dist-test-slave-bxm0 krb5kdc[3443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 21:27:02.668962 3679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:02.638169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43507 (local address 127.0.97.62:33257)
0927 21:27:02.638694 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:02.638704 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:02.640043 (+ 1339us) server_negotiation.cc:407] Connection header received
0927 21:27:02.642880 (+ 2837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:02.642889 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:02.643011 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:02.643142 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:02.645704 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:02.646741 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:02.651804 (+ 5063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:02.652598 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:02.652785 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:02.661869 (+ 9084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:02.661876 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:02.661880 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:02.661944 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:02.665303 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:02.666211 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:02.666216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:02.666218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:02.666357 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:02.666857 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:02.666860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:02.666862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:02.667114 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:02.667221 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:02.668296 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0927 21:27:02.668534 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":228,"threads_started":1}
I20260927 21:27:02.670274 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:02.638950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33257 (local address 127.0.97.1:43507)
0927 21:27:02.639860 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:02.639902 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:02.642475 (+ 2573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:02.643647 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:02.643660 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:02.644544 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:02.645438 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:02.645466 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:02.648974 (+ 3508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:02.648982 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:02.651542 (+ 2560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:02.651564 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:02.652938 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:02.652949 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:02.653385 (+ 436us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:02.658229 (+ 4844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:02.658278 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:02.661447 (+ 3169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:02.665482 (+ 4035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:02.665489 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:02.665505 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:02.666076 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:02.666485 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:02.666489 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:02.666491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:02.666760 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:02.667280 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:02.667291 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:02.667843 (+ 552us) client_negotiation.cc:769] Sending connection context
0927 21:27:02.668163 (+ 320us) client_negotiation.cc:240] Negotiation successful
0927 21:27:02.668807 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":386,"threads_started":1}
I20260927 21:27:02.672019 3677 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33257
I20260927 21:27:02.672386 3677 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:02.673425 3677 heartbeater.cc:511] Master 127.0.97.62:33257 requested a full tablet report, sending...
I20260927 21:27:02.675701 3488 ts_manager.cc:194] Registered new tserver with Master: 9c53b7f796a1448eaa9d661415b59df5 (127.0.97.1:38379)
I20260927 21:27:02.677275 3488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 21:27:02.693367 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33257
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:03.028254 3683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:03.028570 3683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:03.028666 3683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:03.036463 3683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:03.036595 3683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:03.036767 3683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:27:03.047372 3683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:03.048707 3683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:03.050185 3683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:03.063532 3689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:03.063670 3691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:03.064062 3688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:03.064316 3683 server_base.cc:1034] running on GCE node
I20260927 21:27:03.064931 3683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:03.066004 3683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:03.067253 3683 hybrid_clock.cc:648] HybridClock initialized: now 1790544423067164 us; error 70 us; skew 500 ppm
I20260927 21:27:03.067647 3683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:03 dist-test-slave-bxm0 krb5kdc[3443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:03.071228 3683 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 21:27:03.072932 3683 webserver.cc:460] Webserver started at http://127.0.97.2:40625/ using document root <none> and password file <none>
I20260927 21:27:03.073721 3683 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:03.073833 3683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:03.074095 3683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:03.077108 3683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/data/instance:
uuid: "932eb823daac44de8052c183af9a1238"
format_stamp: "Formatted at 2026-09-27 21:27:03 on dist-test-slave-bxm0"
server_key: "d43952ab0a75eb32c7f3a2f8fa3121be"
server_key_iv: "33747868b98aec4a9cba6e25bc516afc"
server_key_version: "encryptionkey@0"
I20260927 21:27:03.078037 3683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/wal/instance:
uuid: "932eb823daac44de8052c183af9a1238"
format_stamp: "Formatted at 2026-09-27 21:27:03 on dist-test-slave-bxm0"
server_key: "d43952ab0a75eb32c7f3a2f8fa3121be"
server_key_iv: "33747868b98aec4a9cba6e25bc516afc"
server_key_version: "encryptionkey@0"
I20260927 21:27:03.083703 3683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:27:03.087616 3698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:03.089174 3683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:27:03.089383 3683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "932eb823daac44de8052c183af9a1238"
format_stamp: "Formatted at 2026-09-27 21:27:03 on dist-test-slave-bxm0"
server_key: "d43952ab0a75eb32c7f3a2f8fa3121be"
server_key_iv: "33747868b98aec4a9cba6e25bc516afc"
server_key_version: "encryptionkey@0"
I20260927 21:27:03.089614 3683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:03.114971 3683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:03.119513 3683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:03.119822 3683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:03.120831 3683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:27:03.122257 3683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:03.122332 3683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:03.122462 3683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:03.122511 3683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:03.157219 3683 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32787
I20260927 21:27:03.157310 3810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32787 every 8 connection(s)
I20260927 21:27:03.158601 3683 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:27:03.164865 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 3683
I20260927 21:27:03.165052 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-1/wal/instance
I20260927 21:27:03.165521 388 external_mini_cluster.cc:1428] Setting key fe137881205fc118edd988d2d01b0b94
Sep 27 21:27:03 dist-test-slave-bxm0 krb5kdc[3443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:27:03.180219 3679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:03.161253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41673 (local address 127.0.97.62:33257)
0927 21:27:03.161462 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:03.161467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:03.162690 (+ 1223us) server_negotiation.cc:407] Connection header received
0927 21:27:03.164580 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:03.164586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:03.164681 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:03.164801 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:03.166697 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:03.167530 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:03.169266 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:03.169818 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:03.169965 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:03.173698 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:03.173704 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:03.173708 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:03.173741 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:03.176502 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:03.177519 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:03.177523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:03.177526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:03.177745 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:03.178270 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:03.178273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:03.178275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:03.178560 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:03.178665 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:03.179675 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0927 21:27:03.179904 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 21:27:03.181875 3813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:03.161791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33257 (local address 127.0.97.2:41673)
0927 21:27:03.162514 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:03.162554 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:03.164251 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:03.165084 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:03.165093 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:03.165767 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:03.166501 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:03.166527 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:03.167687 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:03.167692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:03.169115 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:03.169128 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:03.170068 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:03.170075 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:03.170314 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:03.171054 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:03.171078 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:03.173345 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:03.176661 (+ 3316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:03.176673 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:03.176696 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:03.177379 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:03.177863 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:03.177870 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:03.177877 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:03.178143 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:03.178710 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:03.178723 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:03.179235 (+ 512us) client_negotiation.cc:769] Sending connection context
0927 21:27:03.179577 (+ 342us) client_negotiation.cc:240] Negotiation successful
0927 21:27:03.180200 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":277,"threads_started":1}
I20260927 21:27:03.183509 3811 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33257
I20260927 21:27:03.183863 3811 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 21:27:03.184755 3811 heartbeater.cc:511] Master 127.0.97.62:33257 requested a full tablet report, sending...
I20260927 21:27:03.186374 3488 ts_manager.cc:194] Registered new tserver with Master: 932eb823daac44de8052c183af9a1238 (127.0.97.2:32787)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 21:27:03.187281 3488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 21:27:03.209335 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33257
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:03.553076 3816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:03.553459 3816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:03.553710 3816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:03.561720 3816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:03.561892 3816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:03.562111 3816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:27:03.572448 3816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:03.573940 3816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:03.575346 3816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:03.590498 3824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:03.590519 3822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:03.590761 3821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:03.591420 3816 server_base.cc:1034] running on GCE node
I20260927 21:27:03.592051 3816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:03.593106 3816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:03.594338 3816 hybrid_clock.cc:648] HybridClock initialized: now 1790544423594244 us; error 62 us; skew 500 ppm
I20260927 21:27:03.594667 3816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:03 dist-test-slave-bxm0 krb5kdc[3443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:03.598047 3816 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 21:27:03.599736 3816 webserver.cc:460] Webserver started at http://127.0.97.3:45481/ using document root <none> and password file <none>
I20260927 21:27:03.600446 3816 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:03.600562 3816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:03.600811 3816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:03.603884 3816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/data/instance:
uuid: "4308208c1054405396533a16de63d037"
format_stamp: "Formatted at 2026-09-27 21:27:03 on dist-test-slave-bxm0"
server_key: "543e7767fb6d2ebece93706819536d4f"
server_key_iv: "af16769423bf2075432ef3732a6bc33b"
server_key_version: "encryptionkey@0"
I20260927 21:27:03.604681 3816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4308208c1054405396533a16de63d037"
format_stamp: "Formatted at 2026-09-27 21:27:03 on dist-test-slave-bxm0"
server_key: "543e7767fb6d2ebece93706819536d4f"
server_key_iv: "af16769423bf2075432ef3732a6bc33b"
server_key_version: "encryptionkey@0"
I20260927 21:27:03.610303 3816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 21:27:03.614046 3831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:03.615651 3816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:27:03.615866 3816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "4308208c1054405396533a16de63d037"
format_stamp: "Formatted at 2026-09-27 21:27:03 on dist-test-slave-bxm0"
server_key: "543e7767fb6d2ebece93706819536d4f"
server_key_iv: "af16769423bf2075432ef3732a6bc33b"
server_key_version: "encryptionkey@0"
I20260927 21:27:03.616135 3816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:03.638743 3816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:03.643381 3816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:03.643705 3816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:03.644686 3816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:27:03.646153 3816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:03.646225 3816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:03.646350 3816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:03.646400 3816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:03.681217 3816 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44417
I20260927 21:27:03.681406 3677 heartbeater.cc:503] Master 127.0.97.62:33257 was elected leader, sending a full tablet report...
I20260927 21:27:03.681353 3943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44417 every 8 connection(s)
I20260927 21:27:03.682745 3816 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:27:03.690130 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 3816
I20260927 21:27:03.690320 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790544390748963-388-0/minicluster-data/ts-2/wal/instance
I20260927 21:27:03.690765 388 external_mini_cluster.cc:1428] Setting key 7e145d4dd1470494e4b95a4233794765
Sep 27 21:27:03 dist-test-slave-bxm0 krb5kdc[3443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:27:03.704250 3946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:03.685521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52355 (local address 127.0.97.62:33257)
0927 21:27:03.686025 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:03.686029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:03.687114 (+ 1085us) server_negotiation.cc:407] Connection header received
0927 21:27:03.689016 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:03.689022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:03.689116 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:03.689239 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:03.691035 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:03.692164 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:03.693981 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:03.694379 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:03.694519 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:03.698347 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:03.698353 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:03.698356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:03.698395 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:03.701055 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:03.701897 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:03.701910 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:03.701912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:03.702033 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:03.702471 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:03.702476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:03.702480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:03.702753 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:03.702910 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:03.703636 (+ 726us) server_negotiation.cc:299] Negotiation successful
0927 21:27:03.703855 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":276,"threads_started":1}
I20260927 21:27:03.705338 3947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:03.686115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33257 (local address 127.0.97.3:52355)
0927 21:27:03.686911 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:03.686964 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:03.688709 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:03.689493 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:03.689509 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:03.690102 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:03.690792 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:03.690811 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:03.692337 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:03.692341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:03.693797 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:03.693811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:03.694630 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:03.694638 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:03.694894 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:03.695641 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:03.695670 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:03.698096 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:03.701216 (+ 3120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:03.701226 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:03.701248 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:03.701762 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:03.702147 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:03.702152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:03.702156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:03.702345 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:03.702882 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:03.702891 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:03.703252 (+ 361us) client_negotiation.cc:769] Sending connection context
0927 21:27:03.703534 (+ 282us) client_negotiation.cc:240] Negotiation successful
0927 21:27:03.704092 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":282,"threads_started":1}
I20260927 21:27:03.706933 3944 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33257
I20260927 21:27:03.707223 3944 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:03.707957 3944 heartbeater.cc:511] Master 127.0.97.62:33257 requested a full tablet report, sending...
I20260927 21:27:03.709414 3488 ts_manager.cc:194] Registered new tserver with Master: 4308208c1054405396533a16de63d037 (127.0.97.3:44417)
I20260927 21:27:03.710323 3488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52355
I20260927 21:27:03.719224 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 21:27:03 dist-test-slave-bxm0 krb5kdc[3443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544423, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 21:27:03 dist-test-slave-bxm0 krb5kdc[3443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544423, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:27:03.768079 3946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:03.753815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41838 (local address 127.0.97.62:33257)
0927 21:27:03.754038 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:03.754047 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:03.754564 (+ 517us) server_negotiation.cc:407] Connection header received
0927 21:27:03.754788 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:03.754792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:03.754879 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:03.754980 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:03.756382 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:03.757155 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:03.758522 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:03.758980 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:03.759084 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:03.763027 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:03.763033 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:03.763037 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:03.763074 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:03.765547 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:03.766407 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:03.766411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:03.766413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:03.766550 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:03.766870 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:03.766873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:03.766875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:03.767178 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:03.767360 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:03.767595 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 21:27:03.767763 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 21:27:03.768639 3488 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41838
I20260927 21:27:03.770095 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 3549
Sep 27 21:27:03 dist-test-slave-bxm0 krb5kdc[3443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544423, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 21:27:03.787379 3956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:03.771649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33002 (local address 127.0.97.1:38379)
0927 21:27:03.772262 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:03.772269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:03.772294 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:27:03.772350 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:03.772355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:03.772568 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:03.772721 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:03.773956 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:03.774991 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:03.776267 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:03.776845 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:03.776948 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:03.781169 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:03.781184 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:03.781200 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:03.781238 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:03.784428 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:03.785206 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:03.785213 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:03.785215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:03.785348 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:03.785751 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:03.785755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:03.785757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:03.786152 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:03.786393 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:03.786748 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 21:27:03.787012 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":322,"threads_started":1}
W20260927 21:27:03.788733 3672 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33002
W20260927 21:27:03.789911 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 3549: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:27:03.790165 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 3549
I20260927 21:27:03.806157 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 3683
Sep 27 21:27:03 dist-test-slave-bxm0 krb5kdc[3443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544423, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 21:27:03.822393 3957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:03.807148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51990 (local address 127.0.97.2:32787)
0927 21:27:03.807677 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:03.807684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:03.807708 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:27:03.807765 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:03.807772 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:03.807992 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:03.808184 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:03.809431 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:03.810565 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:03.811861 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:03.812422 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:03.812569 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:03.816797 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:03.816809 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:03.816824 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:03.816869 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:03.819589 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:03.820289 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:03.820297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:03.820299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:03.820441 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:03.820802 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:03.820805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:03.820807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:03.821217 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:03.821459 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:03.821874 (+ 415us) server_negotiation.cc:299] Negotiation successful
0927 21:27:03.822101 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":271,"threads_started":1}
W20260927 21:27:03.823706 3806 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51990
W20260927 21:27:03.824843 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 3683: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:27:03.825037 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 3683
I20260927 21:27:03.840510 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 3816
Sep 27 21:27:03 dist-test-slave-bxm0 krb5kdc[3443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544423, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 21:27:03.858204 3958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:03.841702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52238 (local address 127.0.97.3:44417)
0927 21:27:03.842418 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:03.842425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:03.842454 (+ 29us) server_negotiation.cc:407] Connection header received
0927 21:27:03.842513 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:03.842520 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:03.842736 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:03.842868 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:03.844135 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:03.845208 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:03.847059 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:03.847621 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:03.847727 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:03.852000 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:03.852013 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:03.852028 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:03.852069 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:03.855153 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:03.856006 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:03.856014 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:03.856016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:03.856143 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:03.856541 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:03.856545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:03.856547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:03.856893 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:03.857207 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:03.857540 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 21:27:03.857866 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":421,"threads_started":1}
W20260927 21:27:03.859462 3939 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52238
W20260927 21:27:03.860581 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 3816: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:27:03.860790 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 3816
I20260927 21:27:03.876211 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 3456
I20260927 21:27:03.877394 3518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:03.979002 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 3456
2026-09-27T21:27:03Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3726 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:27:04 dist-test-slave-bxm0 krb5kdc[3962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:27:04 dist-test-slave-bxm0 krb5kdc[3962](info): set up 2 sockets
Sep 27 21:27:04 dist-test-slave-bxm0 krb5kdc[3962](info): commencing operation
krb5kdc: starting...
W20260927 21:27:05.070163 388 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.005s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:27:05 dist-test-slave-bxm0 krb5kdc[3962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:27:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:27:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:27:05.275228 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38593
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:05.603587 3975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:05.603884 3975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:05.604043 3975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:05.611378 3975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:27:05.611508 3975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:05.611603 3975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:05.611711 3975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:27:05.611773 3975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:27:05.622045 3975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:05.623512 3975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:05.624859 3975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:05.638326 3980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:05.638352 3981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:05.638285 3983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:05.639137 3975 server_base.cc:1034] running on GCE node
I20260927 21:27:05.640028 3975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:05.641526 3975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:05.642761 3975 hybrid_clock.cc:648] HybridClock initialized: now 1790544425642692 us; error 67 us; skew 500 ppm
I20260927 21:27:05.643193 3975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:05 dist-test-slave-bxm0 krb5kdc[3962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:05.651444 3975 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:27:05.653200 3975 webserver.cc:460] Webserver started at http://127.0.97.62:45917/ using document root <none> and password file <none>
I20260927 21:27:05.654017 3975 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:05.654137 3975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:05.654423 3975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:05.657732 3975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "db7c170e341b4f59a13eab050b258aca"
format_stamp: "Formatted at 2026-09-27 21:27:05 on dist-test-slave-bxm0"
server_key: "3b97e203af4ebc1ce80fab25af0e1a77"
server_key_iv: "739c52d580950a30a009263b188791d7"
server_key_version: "encryptionkey@0"
I20260927 21:27:05.658543 3975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "db7c170e341b4f59a13eab050b258aca"
format_stamp: "Formatted at 2026-09-27 21:27:05 on dist-test-slave-bxm0"
server_key: "3b97e203af4ebc1ce80fab25af0e1a77"
server_key_iv: "739c52d580950a30a009263b188791d7"
server_key_version: "encryptionkey@0"
I20260927 21:27:05.664112 3975 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:27:05.667809 3990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:05.669342 3975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:27:05.669557 3975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "db7c170e341b4f59a13eab050b258aca"
format_stamp: "Formatted at 2026-09-27 21:27:05 on dist-test-slave-bxm0"
server_key: "3b97e203af4ebc1ce80fab25af0e1a77"
server_key_iv: "739c52d580950a30a009263b188791d7"
server_key_version: "encryptionkey@0"
I20260927 21:27:05.669826 3975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:05.707099 3975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:05.711699 3975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:05.712045 3975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:05.731248 3975 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38593
I20260927 21:27:05.731303 4041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38593 every 8 connection(s)
I20260927 21:27:05.732714 3975 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:27:05.735656 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 3975
I20260927 21:27:05.735916 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:27:05.736537 388 external_mini_cluster.cc:1428] Setting key 11bdc82985649636c225810f8524305d
I20260927 21:27:05.736680 4042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:05.751456 4042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db7c170e341b4f59a13eab050b258aca: Bootstrap starting.
Sep 27 21:27:05 dist-test-slave-bxm0 krb5kdc[3962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:27:05.755635 4042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db7c170e341b4f59a13eab050b258aca: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:05.757066 4042 log.cc:824] T 00000000000000000000000000000000 P db7c170e341b4f59a13eab050b258aca: Log is configured to *not* fsync() on all Append() calls
I20260927 21:27:05.760867 4042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db7c170e341b4f59a13eab050b258aca: No bootstrap required, opened a new log
I20260927 21:27:05.762349 4045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:05.738706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33460 (local address 127.0.97.62:38593)
0927 21:27:05.739542 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:05.739558 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:05.739613 (+ 55us) server_negotiation.cc:407] Connection header received
0927 21:27:05.741720 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:05.741751 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:05.742335 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:05.742828 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:05.744499 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:05.746462 (+ 1963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:05.747815 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:05.749003 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:05.749198 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:05.753175 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:05.753188 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:05.753211 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:05.753272 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:05.757358 (+ 4086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:05.758388 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:05.758396 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:05.758403 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:05.758625 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:05.759084 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:05.759089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:05.759092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:05.759791 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:05.760098 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:05.760522 (+ 424us) server_negotiation.cc:299] Negotiation successful
0927 21:27:05.760982 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":366,"threads_started":1}
I20260927 21:27:05.768967 4042 raft_consensus.cc:348] T 00000000000000000000000000000000 P db7c170e341b4f59a13eab050b258aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7c170e341b4f59a13eab050b258aca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38593 } }
I20260927 21:27:05.769397 4042 raft_consensus.cc:374] T 00000000000000000000000000000000 P db7c170e341b4f59a13eab050b258aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:05.769517 4042 raft_consensus.cc:729] T 00000000000000000000000000000000 P db7c170e341b4f59a13eab050b258aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7c170e341b4f59a13eab050b258aca, State: Initialized, Role: FOLLOWER
I20260927 21:27:05.770268 4042 consensus_queue.cc:260] T 00000000000000000000000000000000 P db7c170e341b4f59a13eab050b258aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7c170e341b4f59a13eab050b258aca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38593 } }
I20260927 21:27:05.770562 4042 raft_consensus.cc:388] T 00000000000000000000000000000000 P db7c170e341b4f59a13eab050b258aca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:27:05.770730 4042 raft_consensus.cc:482] T 00000000000000000000000000000000 P db7c170e341b4f59a13eab050b258aca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:27:05.770955 4042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db7c170e341b4f59a13eab050b258aca [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:05.773135 4042 raft_consensus.cc:504] T 00000000000000000000000000000000 P db7c170e341b4f59a13eab050b258aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7c170e341b4f59a13eab050b258aca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38593 } }
I20260927 21:27:05.773797 4042 leader_election.cc:302] T 00000000000000000000000000000000 P db7c170e341b4f59a13eab050b258aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db7c170e341b4f59a13eab050b258aca; no voters:
I20260927 21:27:05.774521 4042 leader_election.cc:288] T 00000000000000000000000000000000 P db7c170e341b4f59a13eab050b258aca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:27:05.774655 4047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db7c170e341b4f59a13eab050b258aca [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:27:05.775333 4047 raft_consensus.cc:686] T 00000000000000000000000000000000 P db7c170e341b4f59a13eab050b258aca [term 1 LEADER]: Becoming Leader. State: Replica: db7c170e341b4f59a13eab050b258aca, State: Running, Role: LEADER
I20260927 21:27:05.775918 4047 consensus_queue.cc:237] T 00000000000000000000000000000000 P db7c170e341b4f59a13eab050b258aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7c170e341b4f59a13eab050b258aca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38593 } }
I20260927 21:27:05.776930 4042 sys_catalog.cc:564] T 00000000000000000000000000000000 P db7c170e341b4f59a13eab050b258aca [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:27:05.779206 4049 sys_catalog.cc:455] T 00000000000000000000000000000000 P db7c170e341b4f59a13eab050b258aca [sys.catalog]: SysCatalogTable state changed. Reason: New leader db7c170e341b4f59a13eab050b258aca. Latest consensus state: current_term: 1 leader_uuid: "db7c170e341b4f59a13eab050b258aca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7c170e341b4f59a13eab050b258aca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38593 } } }
I20260927 21:27:05.779723 4049 sys_catalog.cc:458] T 00000000000000000000000000000000 P db7c170e341b4f59a13eab050b258aca [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:05.779618 4048 sys_catalog.cc:455] T 00000000000000000000000000000000 P db7c170e341b4f59a13eab050b258aca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db7c170e341b4f59a13eab050b258aca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db7c170e341b4f59a13eab050b258aca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38593 } } }
I20260927 21:27:05.779906 4048 sys_catalog.cc:458] T 00000000000000000000000000000000 P db7c170e341b4f59a13eab050b258aca [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:05.781520 4056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:27:05.786728 4056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:27:05.796271 4056 catalog_manager.cc:1347] Generated new cluster ID: c1f2e2fefa69492d94d1cf2da697ae32
I20260927 21:27:05.796411 4056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:27:05.817495 4056 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:27:05.819568 4056 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:27:05.834124 4056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db7c170e341b4f59a13eab050b258aca: Generated new TSK 0
I20260927 21:27:05.835251 4056 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 21:27:05.889846 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38593
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:06.227855 4068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:06.228168 4068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:06.228334 4068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:06.235803 4068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:06.235929 4068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:06.236124 4068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:27:06.246425 4068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:06.247901 4068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:06.249245 4068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:06.262861 4073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:06.262938 4074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:06.262964 4076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:06.264161 4068 server_base.cc:1034] running on GCE node
I20260927 21:27:06.264849 4068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:06.265959 4068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:06.267228 4068 hybrid_clock.cc:648] HybridClock initialized: now 1790544426267152 us; error 73 us; skew 500 ppm
I20260927 21:27:06.267578 4068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:06 dist-test-slave-bxm0 krb5kdc[3962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:06.271014 4068 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 21:27:06.272470 4068 webserver.cc:460] Webserver started at http://127.0.97.1:38167/ using document root <none> and password file <none>
I20260927 21:27:06.273176 4068 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:06.273276 4068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:06.273530 4068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:06.276577 4068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-0/data/instance:
uuid: "296261562b754ee38952e5f76a7c9a8c"
format_stamp: "Formatted at 2026-09-27 21:27:06 on dist-test-slave-bxm0"
server_key: "8eae6027319ede8e49db4a5f2e9e761d"
server_key_iv: "f677c2dbbf4b2f1701e9815699368282"
server_key_version: "encryptionkey@0"
I20260927 21:27:06.277420 4068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-0/wal/instance:
uuid: "296261562b754ee38952e5f76a7c9a8c"
format_stamp: "Formatted at 2026-09-27 21:27:06 on dist-test-slave-bxm0"
server_key: "8eae6027319ede8e49db4a5f2e9e761d"
server_key_iv: "f677c2dbbf4b2f1701e9815699368282"
server_key_version: "encryptionkey@0"
I20260927 21:27:06.283121 4068 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 21:27:06.286670 4083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:06.288127 4068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:27:06.288318 4068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "296261562b754ee38952e5f76a7c9a8c"
format_stamp: "Formatted at 2026-09-27 21:27:06 on dist-test-slave-bxm0"
server_key: "8eae6027319ede8e49db4a5f2e9e761d"
server_key_iv: "f677c2dbbf4b2f1701e9815699368282"
server_key_version: "encryptionkey@0"
I20260927 21:27:06.288550 4068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:06.305006 4068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:06.309595 4068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:06.309876 4068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:06.310787 4068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:27:06.312084 4068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:06.312156 4068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:06.312237 4068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:06.312286 4068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:06.346056 4068 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33853
I20260927 21:27:06.346169 4195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33853 every 8 connection(s)
I20260927 21:27:06.347370 4068 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:27:06.350057 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 4068
I20260927 21:27:06.350296 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-0/wal/instance
I20260927 21:27:06.351358 388 external_mini_cluster.cc:1428] Setting key a4844a0d1bb4f4a463f1607504b45c37
Sep 27 21:27:06 dist-test-slave-bxm0 krb5kdc[3962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:27:06.371271 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:06.350230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35317 (local address 127.0.97.62:38593)
0927 21:27:06.352553 (+ 2323us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:06.352558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:06.352577 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:27:06.354078 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:06.354085 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:06.354180 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:06.354321 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:06.355813 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:06.357117 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:06.358958 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:06.359414 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:06.359527 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:06.363793 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:06.363802 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:06.363811 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:06.363859 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:06.367619 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:06.368570 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:06.368580 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:06.368586 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:06.368750 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:06.369232 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:06.369239 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:06.369246 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:06.369684 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:06.369834 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:06.370601 (+ 767us) server_negotiation.cc:299] Negotiation successful
0927 21:27:06.370849 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2215,"thread_start_us":2066,"threads_started":1}
I20260927 21:27:06.371918 4199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:06.350769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38593 (local address 127.0.97.1:35317)
0927 21:27:06.351641 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:06.351687 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:06.353783 (+ 2096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:06.354529 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:06.354537 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:06.355045 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:06.355620 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:06.355638 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:06.357292 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:06.357296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:06.358793 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:06.358805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:06.359675 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:06.359684 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:06.360040 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:06.360906 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:06.360936 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:06.363525 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:06.367864 (+ 4339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:06.367877 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:06.367900 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:06.368384 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:06.368885 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:06.368889 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:06.368891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:06.369098 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:06.369818 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:06.369824 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:06.370172 (+ 348us) client_negotiation.cc:769] Sending connection context
0927 21:27:06.370399 (+ 227us) client_negotiation.cc:240] Negotiation successful
0927 21:27:06.370804 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":395,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 21:27:06.373816 4196 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38593
I20260927 21:27:06.374173 4196 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 21:27:06.375157 4196 heartbeater.cc:511] Master 127.0.97.62:38593 requested a full tablet report, sending...
I20260927 21:27:06.377817 4007 ts_manager.cc:194] Registered new tserver with Master: 296261562b754ee38952e5f76a7c9a8c (127.0.97.1:33853)
I20260927 21:27:06.380056 4007 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35317
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 21:27:06.401523 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38593
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:06.733322 4202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:06.733721 4202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:06.733901 4202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:06.741442 4202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:06.741613 4202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:06.741853 4202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:27:06.752604 4202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:06.754089 4202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:06.755435 4202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:06.769052 4207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:06.769042 4208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:06.769083 4210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:06.769984 4202 server_base.cc:1034] running on GCE node
I20260927 21:27:06.770658 4202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:06.771586 4202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:06.772830 4202 hybrid_clock.cc:648] HybridClock initialized: now 1790544426772784 us; error 44 us; skew 500 ppm
I20260927 21:27:06.773190 4202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:06 dist-test-slave-bxm0 krb5kdc[3962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:06.776566 4202 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 21:27:06.778424 4202 webserver.cc:460] Webserver started at http://127.0.97.2:34229/ using document root <none> and password file <none>
I20260927 21:27:06.779119 4202 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:06.779222 4202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:06.779474 4202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:06.782835 4202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-1/data/instance:
uuid: "6d8068c88b6a4db58bf8f3d9f119be8b"
format_stamp: "Formatted at 2026-09-27 21:27:06 on dist-test-slave-bxm0"
server_key: "24ea5893b46a5bcd0b3d40118a32dc6f"
server_key_iv: "8e7f59cd8eb3290b5deb2c5534bf9717"
server_key_version: "encryptionkey@0"
I20260927 21:27:06.783619 4202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6d8068c88b6a4db58bf8f3d9f119be8b"
format_stamp: "Formatted at 2026-09-27 21:27:06 on dist-test-slave-bxm0"
server_key: "24ea5893b46a5bcd0b3d40118a32dc6f"
server_key_iv: "8e7f59cd8eb3290b5deb2c5534bf9717"
server_key_version: "encryptionkey@0"
I20260927 21:27:06.789155 4202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:27:06.793048 4217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:06.794636 4202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:27:06.794792 4202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "6d8068c88b6a4db58bf8f3d9f119be8b"
format_stamp: "Formatted at 2026-09-27 21:27:06 on dist-test-slave-bxm0"
server_key: "24ea5893b46a5bcd0b3d40118a32dc6f"
server_key_iv: "8e7f59cd8eb3290b5deb2c5534bf9717"
server_key_version: "encryptionkey@0"
I20260927 21:27:06.794983 4202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:06.814711 4202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:06.819114 4202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:06.819375 4202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:06.820362 4202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:27:06.821838 4202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:06.821910 4202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:06.822006 4202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:06.822052 4202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:06.855696 4202 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38601
I20260927 21:27:06.855746 4329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38601 every 8 connection(s)
I20260927 21:27:06.856985 4202 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:27:06.861829 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 4202
I20260927 21:27:06.862047 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-1/wal/instance
I20260927 21:27:06.862699 388 external_mini_cluster.cc:1428] Setting key 0ec072b99e4071e721176a3ba018f645
Sep 27 21:27:06 dist-test-slave-bxm0 krb5kdc[3962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:27:06.882273 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:06.859816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44155 (local address 127.0.97.62:38593)
0927 21:27:06.860026 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:06.860033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:06.861328 (+ 1295us) server_negotiation.cc:407] Connection header received
0927 21:27:06.863984 (+ 2656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:06.863989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:06.864084 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:06.864197 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:06.866374 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:06.867065 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:06.869714 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:06.870150 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:06.870296 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:06.874305 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:06.874315 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:06.874319 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:06.874368 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:06.878138 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:06.879207 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:06.879215 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:06.879221 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:06.879397 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:06.880014 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:06.880021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:06.880027 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:06.880505 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:06.880686 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:06.881550 (+ 864us) server_negotiation.cc:299] Negotiation successful
0927 21:27:06.881847 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 21:27:06.883288 4332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:06.860342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38593 (local address 127.0.97.2:44155)
0927 21:27:06.861141 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:06.861182 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:06.863657 (+ 2475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:06.864488 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:06.864499 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:06.865261 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:06.866171 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:06.866200 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:06.867231 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:06.867240 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:06.869489 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:06.869507 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:06.870413 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:06.870419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:06.870666 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:06.871416 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:06.871445 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:06.873988 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:06.878400 (+ 4412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:06.878411 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:06.878426 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:06.879036 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:06.879548 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:06.879552 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:06.879555 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:06.879829 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:06.880660 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:06.880667 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:06.881106 (+ 439us) client_negotiation.cc:769] Sending connection context
0927 21:27:06.881426 (+ 320us) client_negotiation.cc:240] Negotiation successful
0927 21:27:06.881983 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":317,"threads_started":1}
I20260927 21:27:06.884914 4330 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38593
I20260927 21:27:06.885237 4330 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 21:27:06.886096 4330 heartbeater.cc:511] Master 127.0.97.62:38593 requested a full tablet report, sending...
I20260927 21:27:06.887738 4007 ts_manager.cc:194] Registered new tserver with Master: 6d8068c88b6a4db58bf8f3d9f119be8b (127.0.97.2:38601)
I20260927 21:27:06.888988 4007 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 21:27:06.910641 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38593
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:07.252935 4335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:07.253249 4335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:07.253345 4335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:07.261921 4335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:07.262055 4335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:07.262177 4335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:27:07.272435 4335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:07.274089 4335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:07.275489 4335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:07.289920 4341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:07.290108 4340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:07.290238 4343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:07.291350 4335 server_base.cc:1034] running on GCE node
I20260927 21:27:07.292022 4335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:07.293136 4335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:07.294363 4335 hybrid_clock.cc:648] HybridClock initialized: now 1790544427294281 us; error 80 us; skew 500 ppm
I20260927 21:27:07.294698 4335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:07 dist-test-slave-bxm0 krb5kdc[3962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:07.298045 4335 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 21:27:07.299823 4335 webserver.cc:460] Webserver started at http://127.0.97.3:33847/ using document root <none> and password file <none>
I20260927 21:27:07.300551 4335 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:07.300700 4335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:07.301026 4335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:07.304127 4335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-2/data/instance:
uuid: "834aab658b704650ab4d9101f701e993"
format_stamp: "Formatted at 2026-09-27 21:27:07 on dist-test-slave-bxm0"
server_key: "5dbdd935e63200b1be8256009f999faa"
server_key_iv: "f4cb4d5dd237807215bc88e1747276d7"
server_key_version: "encryptionkey@0"
I20260927 21:27:07.305033 4335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-2/wal/instance:
uuid: "834aab658b704650ab4d9101f701e993"
format_stamp: "Formatted at 2026-09-27 21:27:07 on dist-test-slave-bxm0"
server_key: "5dbdd935e63200b1be8256009f999faa"
server_key_iv: "f4cb4d5dd237807215bc88e1747276d7"
server_key_version: "encryptionkey@0"
I20260927 21:27:07.310680 4335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 21:27:07.314653 4350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:07.316226 4335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:27:07.316425 4335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "834aab658b704650ab4d9101f701e993"
format_stamp: "Formatted at 2026-09-27 21:27:07 on dist-test-slave-bxm0"
server_key: "5dbdd935e63200b1be8256009f999faa"
server_key_iv: "f4cb4d5dd237807215bc88e1747276d7"
server_key_version: "encryptionkey@0"
I20260927 21:27:07.316664 4335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:07.341197 4335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:07.345742 4335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:07.346066 4335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:07.347265 4335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:27:07.348971 4335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:07.349076 4335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:07.349195 4335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:07.349257 4335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:07.384055 4196 heartbeater.cc:503] Master 127.0.97.62:38593 was elected leader, sending a full tablet report...
I20260927 21:27:07.384373 4335 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38751
I20260927 21:27:07.384462 4462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38751 every 8 connection(s)
I20260927 21:27:07.386044 4335 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:27:07.391301 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 4335
I20260927 21:27:07.391512 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790544390748963-388-0/minicluster-data/ts-2/wal/instance
I20260927 21:27:07.392073 388 external_mini_cluster.cc:1428] Setting key 7797f31fcc182a9b94a87c2ab5b3b580
Sep 27 21:27:07 dist-test-slave-bxm0 krb5kdc[3962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:27:07.408030 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:07.388612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47349 (local address 127.0.97.62:38593)
0927 21:27:07.389056 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:07.389063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:07.390347 (+ 1284us) server_negotiation.cc:407] Connection header received
0927 21:27:07.393246 (+ 2899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:07.393250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:07.393312 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:07.393447 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:07.395310 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:07.396088 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:07.397707 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:07.398163 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:07.398294 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:07.402072 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:07.402077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:07.402081 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:07.402117 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:07.404626 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:07.405366 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:07.405369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:07.405371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:07.405490 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:07.406060 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:07.406064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:07.406067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:07.406404 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:07.406563 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:07.407351 (+ 788us) server_negotiation.cc:299] Negotiation successful
0927 21:27:07.407607 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":207,"threads_started":1}
I20260927 21:27:07.409055 4466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:07.389305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38593 (local address 127.0.97.3:47349)
0927 21:27:07.390170 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:07.390218 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:07.392580 (+ 2362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:07.393806 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:07.393820 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:07.394508 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:07.395137 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:07.395155 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:07.396245 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:07.396251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:07.397516 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:07.397527 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:07.398403 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:07.398410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:07.398652 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:07.399387 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:07.399411 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:07.401805 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:07.404773 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:07.404780 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:07.404795 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:07.405250 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:07.405649 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:07.405654 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:07.405658 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:07.405913 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:07.406541 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:07.406549 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:07.406965 (+ 416us) client_negotiation.cc:769] Sending connection context
0927 21:27:07.407247 (+ 282us) client_negotiation.cc:240] Negotiation successful
0927 21:27:07.407827 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":386,"threads_started":1}
I20260927 21:27:07.410535 4463 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38593
I20260927 21:27:07.410827 4463 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:07.411590 4463 heartbeater.cc:511] Master 127.0.97.62:38593 requested a full tablet report, sending...
I20260927 21:27:07.413091 4007 ts_manager.cc:194] Registered new tserver with Master: 834aab658b704650ab4d9101f701e993 (127.0.97.3:38751)
I20260927 21:27:07.414009 4007 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47349
I20260927 21:27:07.420464 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 21:27:07 dist-test-slave-bxm0 krb5kdc[3962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:27:07 dist-test-slave-bxm0 krb5kdc[3962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 21:27:07.471942 4471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:07.455772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52962 (local address 127.0.97.1:33853)
0927 21:27:07.456437 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:07.456448 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:07.456485 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:27:07.456570 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:07.456576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:07.456791 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:07.456971 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:07.458487 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:07.459552 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:07.460851 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:07.461448 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:07.461558 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:07.465864 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:07.465884 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:07.465913 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:07.465962 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:07.468997 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:07.469830 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:07.469838 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:07.469840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:07.470034 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:07.470457 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:07.470461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:07.470464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:07.470827 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:07.471085 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:07.471409 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 21:27:07.471622 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":292,"threads_started":1}
Sep 27 21:27:07 dist-test-slave-bxm0 krb5kdc[3962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:27:07.491300 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:07.476583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33468 (local address 127.0.97.62:38593)
0927 21:27:07.476810 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:07.476814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:07.477217 (+ 403us) server_negotiation.cc:407] Connection header received
0927 21:27:07.477417 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:07.477423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:07.477509 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:07.477619 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:07.478992 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:07.479728 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:07.481192 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:07.481592 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:07.481752 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:07.485948 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:07.485954 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:07.485958 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:07.486003 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:07.488694 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:07.489347 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:07.489351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:07.489353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:07.489487 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:07.489887 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:07.489890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:07.489892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:07.490207 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:07.490402 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:07.490811 (+ 409us) server_negotiation.cc:299] Negotiation successful
0927 21:27:07.491030 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260927 21:27:07.492026 4007 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33468
I20260927 21:27:07.493252 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 4068
I20260927 21:27:07.506397 4471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:07.494388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52966 (local address 127.0.97.1:33853)
0927 21:27:07.494611 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:07.494617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:07.494830 (+ 213us) server_negotiation.cc:407] Connection header received
0927 21:27:07.495048 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:07.495053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:07.495122 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:07.495226 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:07.496463 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:07.497148 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:07.498500 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:07.498928 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:07.499094 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:07.501318 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:07.501323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:07.501326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:07.501360 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:07.503825 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:07.504493 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:07.504500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:07.504503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:07.504632 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:07.504993 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:07.504999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:07.505003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:07.505254 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:07.505513 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:07.505836 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 21:27:07.506066 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 21:27:07.507086 4191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:07.590857 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 4068
I20260927 21:27:07.607467 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 4202
Sep 27 21:27:07 dist-test-slave-bxm0 krb5kdc[3962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 21:27:07.625344 4476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:07.608419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36068 (local address 127.0.97.2:38601)
0927 21:27:07.609010 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:07.609017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:07.609054 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:27:07.609124 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:07.609131 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:07.609339 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:07.609533 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:07.610765 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:07.611815 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:07.613124 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:07.613781 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:07.613923 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:07.618223 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:07.618237 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:07.618253 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:07.618300 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:07.621328 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:07.622367 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:07.622379 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:07.622385 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:07.622565 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:07.623112 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:07.623119 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:07.623124 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:07.623635 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:07.623990 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:07.624461 (+ 471us) server_negotiation.cc:299] Negotiation successful
0927 21:27:07.624813 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":273,"threads_started":1}
I20260927 21:27:07.627058 4325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:07.716027 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 4202
I20260927 21:27:07.733280 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 4335
Sep 27 21:27:07 dist-test-slave-bxm0 krb5kdc[3962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 21:27:07.753891 4478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:07.734527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49620 (local address 127.0.97.3:38751)
0927 21:27:07.735383 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:07.735391 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:07.735431 (+ 40us) server_negotiation.cc:407] Connection header received
0927 21:27:07.735490 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:07.735496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:07.735720 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:07.735855 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:07.737559 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:07.738712 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:07.740660 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:07.741285 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:07.741405 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:07.746534 (+ 5129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:07.746559 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:07.746576 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:07.746637 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:07.750556 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:07.751447 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:07.751458 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:07.751461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:07.751622 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:07.752101 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:07.752106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:07.752108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:07.752478 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:07.752763 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:07.753077 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 21:27:07.753309 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":407,"threads_started":1}
I20260927 21:27:07.755472 4458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:07.838747 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 4335
I20260927 21:27:07.854775 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 3975
I20260927 21:27:07.856179 4037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:07.957830 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 3975
2026-09-27T21:27:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3978 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:27:08 dist-test-slave-bxm0 krb5kdc[4483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:27:08 dist-test-slave-bxm0 krb5kdc[4483](info): set up 2 sockets
Sep 27 21:27:08 dist-test-slave-bxm0 krb5kdc[4483](info): commencing operation
krb5kdc: starting...
W20260927 21:27:09.078184 388 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:27:09 dist-test-slave-bxm0 krb5kdc[4483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:27:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:27:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:27:09.286049 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35003
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:09.614566 4496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:09.614879 4496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:09.615078 4496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:09.624303 4496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:27:09.624486 4496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:09.624583 4496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:09.624671 4496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:27:09.624749 4496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:27:09.637272 4496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:09.638682 4496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:09.639958 4496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:09.652127 4501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:09.652130 4504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:09.652108 4502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:09.653108 4496 server_base.cc:1034] running on GCE node
I20260927 21:27:09.654023 4496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:09.655588 4496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:09.656873 4496 hybrid_clock.cc:648] HybridClock initialized: now 1790544429656833 us; error 55 us; skew 500 ppm
I20260927 21:27:09.657294 4496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:09 dist-test-slave-bxm0 krb5kdc[4483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:09.660723 4496 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:27:09.662166 4496 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:09.662310 4496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:09.662676 4496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:09.666011 4496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "2ad0c6c8444d4452972678ca04dabeb6"
format_stamp: "Formatted at 2026-09-27 21:27:09 on dist-test-slave-bxm0"
server_key: "aab9c90eef2bbc50a90038d04d5ef3bf"
server_key_iv: "2eec67862efa17238d5b5f4ba1aa14a4"
server_key_version: "encryptionkey@0"
I20260927 21:27:09.666985 4496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "2ad0c6c8444d4452972678ca04dabeb6"
format_stamp: "Formatted at 2026-09-27 21:27:09 on dist-test-slave-bxm0"
server_key: "aab9c90eef2bbc50a90038d04d5ef3bf"
server_key_iv: "2eec67862efa17238d5b5f4ba1aa14a4"
server_key_version: "encryptionkey@0"
I20260927 21:27:09.672586 4496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:27:09.676573 4510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:09.678095 4496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:27:09.678310 4496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "2ad0c6c8444d4452972678ca04dabeb6"
format_stamp: "Formatted at 2026-09-27 21:27:09 on dist-test-slave-bxm0"
server_key: "aab9c90eef2bbc50a90038d04d5ef3bf"
server_key_iv: "2eec67862efa17238d5b5f4ba1aa14a4"
server_key_version: "encryptionkey@0"
I20260927 21:27:09.678565 4496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:09.702798 4496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:09.707376 4496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:09.707697 4496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:09.726914 4496 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35003
I20260927 21:27:09.726958 4561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35003 every 8 connection(s)
I20260927 21:27:09.727855 4496 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:27:09.732025 4562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:09.737349 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 4496
I20260927 21:27:09.737648 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:27:09.738343 388 external_mini_cluster.cc:1428] Setting key 8093e324c501967a832a12fa6774d995
I20260927 21:27:09.745347 4562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad0c6c8444d4452972678ca04dabeb6: Bootstrap starting.
I20260927 21:27:09.749696 4562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ad0c6c8444d4452972678ca04dabeb6: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:09.750901 4562 log.cc:824] T 00000000000000000000000000000000 P 2ad0c6c8444d4452972678ca04dabeb6: Log is configured to *not* fsync() on all Append() calls
I20260927 21:27:09.754282 4562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad0c6c8444d4452972678ca04dabeb6: No bootstrap required, opened a new log
Sep 27 21:27:09 dist-test-slave-bxm0 krb5kdc[4483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:27:09.765940 4562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ad0c6c8444d4452972678ca04dabeb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad0c6c8444d4452972678ca04dabeb6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35003 } }
I20260927 21:27:09.766474 4562 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ad0c6c8444d4452972678ca04dabeb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:09.766601 4562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ad0c6c8444d4452972678ca04dabeb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ad0c6c8444d4452972678ca04dabeb6, State: Initialized, Role: FOLLOWER
I20260927 21:27:09.767025 4565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:09.740675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40652 (local address 127.0.97.62:35003)
0927 21:27:09.741736 (+ 1061us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:09.741755 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:09.741819 (+ 64us) server_negotiation.cc:407] Connection header received
0927 21:27:09.744466 (+ 2647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:09.744500 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:09.745067 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:09.745596 (+ 529us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:09.747140 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:09.749232 (+ 2092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:09.750862 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:09.752018 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:09.752186 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:09.756593 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:09.756606 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:09.756641 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:09.756697 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:09.761306 (+ 4609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:09.762251 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:09.762273 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:09.762285 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:09.762565 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:09.763053 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:09.763058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:09.763062 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:09.764034 (+ 972us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:09.764394 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:09.764843 (+ 449us) server_negotiation.cc:299] Negotiation successful
0927 21:27:09.765338 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":467,"threads_started":1}
I20260927 21:27:09.767433 4562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ad0c6c8444d4452972678ca04dabeb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad0c6c8444d4452972678ca04dabeb6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35003 } }
I20260927 21:27:09.767725 4562 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ad0c6c8444d4452972678ca04dabeb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:27:09.767879 4562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ad0c6c8444d4452972678ca04dabeb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:27:09.768095 4562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ad0c6c8444d4452972678ca04dabeb6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:09.770948 4562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ad0c6c8444d4452972678ca04dabeb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad0c6c8444d4452972678ca04dabeb6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35003 } }
I20260927 21:27:09.771530 4562 leader_election.cc:302] T 00000000000000000000000000000000 P 2ad0c6c8444d4452972678ca04dabeb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ad0c6c8444d4452972678ca04dabeb6; no voters:
I20260927 21:27:09.772176 4562 leader_election.cc:288] T 00000000000000000000000000000000 P 2ad0c6c8444d4452972678ca04dabeb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:27:09.772347 4567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ad0c6c8444d4452972678ca04dabeb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:27:09.773054 4567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ad0c6c8444d4452972678ca04dabeb6 [term 1 LEADER]: Becoming Leader. State: Replica: 2ad0c6c8444d4452972678ca04dabeb6, State: Running, Role: LEADER
I20260927 21:27:09.773600 4567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ad0c6c8444d4452972678ca04dabeb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad0c6c8444d4452972678ca04dabeb6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35003 } }
I20260927 21:27:09.774590 4562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ad0c6c8444d4452972678ca04dabeb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:27:09.776551 4568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad0c6c8444d4452972678ca04dabeb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ad0c6c8444d4452972678ca04dabeb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad0c6c8444d4452972678ca04dabeb6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35003 } } }
I20260927 21:27:09.777040 4568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad0c6c8444d4452972678ca04dabeb6 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:09.777189 4569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad0c6c8444d4452972678ca04dabeb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ad0c6c8444d4452972678ca04dabeb6. Latest consensus state: current_term: 1 leader_uuid: "2ad0c6c8444d4452972678ca04dabeb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad0c6c8444d4452972678ca04dabeb6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35003 } } }
I20260927 21:27:09.777522 4569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad0c6c8444d4452972678ca04dabeb6 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:09.778649 4572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:27:09.784696 4572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:27:09.793708 4572 catalog_manager.cc:1347] Generated new cluster ID: 5f3197c477f7497ca3b2735defe2f46e
I20260927 21:27:09.793848 4572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:27:09.820719 4572 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:27:09.822206 4572 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:27:09.832402 4572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ad0c6c8444d4452972678ca04dabeb6: Generated new TSK 0
I20260927 21:27:09.833372 4572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 21:27:09.889674 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35003
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:10.227835 4588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:10.228139 4588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:10.228302 4588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:10.235869 4588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:10.235994 4588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:10.236195 4588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:27:10.246289 4588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:10.247737 4588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:10.249151 4588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:10.262993 4593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:10.262957 4594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:10.262914 4596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:10.263972 4588 server_base.cc:1034] running on GCE node
I20260927 21:27:10.264689 4588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:10.266083 4588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:10.267364 4588 hybrid_clock.cc:648] HybridClock initialized: now 1790544430267302 us; error 53 us; skew 500 ppm
I20260927 21:27:10.267752 4588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:10 dist-test-slave-bxm0 krb5kdc[4483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:10.271464 4588 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 21:27:10.272791 4588 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:10.272895 4588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:10.273314 4588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:10.276762 4588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-0/data/instance:
uuid: "5ce2c7e47cd44c7096b5841c60f946e5"
format_stamp: "Formatted at 2026-09-27 21:27:10 on dist-test-slave-bxm0"
server_key: "e5778517d5aada04728da10bda6b2385"
server_key_iv: "5f8b3731506c8bd02fa0718104478f51"
server_key_version: "encryptionkey@0"
I20260927 21:27:10.277755 4588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5ce2c7e47cd44c7096b5841c60f946e5"
format_stamp: "Formatted at 2026-09-27 21:27:10 on dist-test-slave-bxm0"
server_key: "e5778517d5aada04728da10bda6b2385"
server_key_iv: "5f8b3731506c8bd02fa0718104478f51"
server_key_version: "encryptionkey@0"
I20260927 21:27:10.283859 4588 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 21:27:10.287798 4602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:10.289415 4588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:27:10.289602 4588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "5ce2c7e47cd44c7096b5841c60f946e5"
format_stamp: "Formatted at 2026-09-27 21:27:10 on dist-test-slave-bxm0"
server_key: "e5778517d5aada04728da10bda6b2385"
server_key_iv: "5f8b3731506c8bd02fa0718104478f51"
server_key_version: "encryptionkey@0"
I20260927 21:27:10.289909 4588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:10.307113 4588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:10.311406 4588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:10.311650 4588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:10.312609 4588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:27:10.313987 4588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:10.314075 4588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:10.314162 4588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:10.314216 4588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:10.349416 4588 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40071
I20260927 21:27:10.349519 4714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40071 every 8 connection(s)
I20260927 21:27:10.350353 4588 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:27:10.360397 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 4588
I20260927 21:27:10.360627 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-0/wal/instance
I20260927 21:27:10.361294 388 external_mini_cluster.cc:1428] Setting key cf5daf3dff80f02e58a78b21f04109af
Sep 27 21:27:10 dist-test-slave-bxm0 krb5kdc[4483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:27:10.374341 4717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:10.352934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34391 (local address 127.0.97.62:35003)
0927 21:27:10.353409 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:10.353414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:10.354478 (+ 1064us) server_negotiation.cc:407] Connection header received
0927 21:27:10.356952 (+ 2474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:10.356957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:10.357076 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:10.357213 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:10.358920 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:10.359722 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:10.361852 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:10.362510 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:10.362669 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:10.367206 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:10.367212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:10.367215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:10.367252 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:10.370524 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:10.371482 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:10.371491 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:10.371498 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:10.371651 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:10.372148 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:10.372156 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:10.372163 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:10.372572 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:10.372741 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:10.373539 (+ 798us) server_negotiation.cc:299] Negotiation successful
0927 21:27:10.373843 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":228,"threads_started":1}
I20260927 21:27:10.375231 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:10.353530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35003 (local address 127.0.97.1:34391)
0927 21:27:10.354304 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:10.354344 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:10.356641 (+ 2297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:10.357478 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:10.357487 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:10.358114 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:10.358728 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:10.358747 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:10.359915 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:10.359920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:10.361590 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:10.361605 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:10.362760 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:10.362766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:10.363011 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:10.363816 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:10.363845 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:10.366872 (+ 3027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:10.370710 (+ 3838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:10.370718 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:10.370734 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:10.371314 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:10.371775 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:10.371779 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:10.371781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:10.372018 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:10.372741 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:10.372748 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:10.373125 (+ 377us) client_negotiation.cc:769] Sending connection context
0927 21:27:10.373380 (+ 255us) client_negotiation.cc:240] Negotiation successful
0927 21:27:10.373929 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":294,"threads_started":1}
I20260927 21:27:10.377116 4715 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35003
I20260927 21:27:10.377477 4715 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:10.378412 4715 heartbeater.cc:511] Master 127.0.97.62:35003 requested a full tablet report, sending...
I20260927 21:27:10.380918 4527 ts_manager.cc:194] Registered new tserver with Master: 5ce2c7e47cd44c7096b5841c60f946e5 (127.0.97.1:40071)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 21:27:10.382740 4527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34391
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 21:27:10.409686 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35003
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:10.740595 4721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:10.740942 4721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:10.741117 4721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:10.748920 4721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:10.749086 4721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:10.749261 4721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:27:10.759538 4721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:10.761287 4721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:10.762728 4721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:10.776085 4726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:10.776124 4727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:10.776021 4729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:10.777136 4721 server_base.cc:1034] running on GCE node
I20260927 21:27:10.777930 4721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:10.778985 4721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:10.780257 4721 hybrid_clock.cc:648] HybridClock initialized: now 1790544430780149 us; error 66 us; skew 500 ppm
I20260927 21:27:10.780658 4721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:10 dist-test-slave-bxm0 krb5kdc[4483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:10.787804 4721 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 21:27:10.789104 4721 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:10.789202 4721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:10.789476 4721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:10.792513 4721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-1/data/instance:
uuid: "8f069afb2fa0402c97bf96231fee0e4d"
format_stamp: "Formatted at 2026-09-27 21:27:10 on dist-test-slave-bxm0"
server_key: "d83945ac360f49bfafb3c33ab2e1a953"
server_key_iv: "3258c505a2b57ed6024ad8cb0d545c0e"
server_key_version: "encryptionkey@0"
I20260927 21:27:10.793346 4721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8f069afb2fa0402c97bf96231fee0e4d"
format_stamp: "Formatted at 2026-09-27 21:27:10 on dist-test-slave-bxm0"
server_key: "d83945ac360f49bfafb3c33ab2e1a953"
server_key_iv: "3258c505a2b57ed6024ad8cb0d545c0e"
server_key_version: "encryptionkey@0"
I20260927 21:27:10.799036 4721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260927 21:27:10.802788 4735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:10.804314 4721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260927 21:27:10.804535 4721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "8f069afb2fa0402c97bf96231fee0e4d"
format_stamp: "Formatted at 2026-09-27 21:27:10 on dist-test-slave-bxm0"
server_key: "d83945ac360f49bfafb3c33ab2e1a953"
server_key_iv: "3258c505a2b57ed6024ad8cb0d545c0e"
server_key_version: "encryptionkey@0"
I20260927 21:27:10.804795 4721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:10.818562 4721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:10.823180 4721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:10.823494 4721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:10.824505 4721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:27:10.826005 4721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:10.826124 4721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:10.826264 4721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:10.826362 4721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:10.860414 4721 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33023
I20260927 21:27:10.860472 4847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33023 every 8 connection(s)
I20260927 21:27:10.861325 4721 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:27:10.870004 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 4721
I20260927 21:27:10.870216 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-1/wal/instance
I20260927 21:27:10.870648 388 external_mini_cluster.cc:1428] Setting key f2136f861c2563958599e91098cb8379
Sep 27 21:27:10 dist-test-slave-bxm0 krb5kdc[4483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:27:10.882097 4717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:10.863928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55215 (local address 127.0.97.62:35003)
0927 21:27:10.864118 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:10.864123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:10.865308 (+ 1185us) server_negotiation.cc:407] Connection header received
0927 21:27:10.867186 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:10.867191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:10.867263 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:10.867359 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:10.868842 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:10.869667 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:10.871598 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:10.872089 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:10.872223 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:10.876196 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:10.876201 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:10.876205 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:10.876239 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:10.878839 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:10.879639 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:10.879647 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:10.879651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:10.879789 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:10.880229 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:10.880235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:10.880239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:10.880590 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:10.880741 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:10.881450 (+ 709us) server_negotiation.cc:299] Negotiation successful
0927 21:27:10.881717 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":72}
I20260927 21:27:10.883287 4850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:10.864460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35003 (local address 127.0.97.2:55215)
0927 21:27:10.865148 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:10.865186 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:10.866904 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:10.867585 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:10.867594 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:10.868099 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:10.868667 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:10.868684 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:10.869873 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:10.869877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:10.871454 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:10.871467 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:10.872328 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:10.872335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:10.872566 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:10.873297 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:10.873324 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:10.875935 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:10.879026 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:10.879036 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:10.879054 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:10.879509 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:10.879908 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:10.879915 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:10.879919 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:10.880113 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:10.880715 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:10.880724 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:10.881088 (+ 364us) client_negotiation.cc:769] Sending connection context
0927 21:27:10.881372 (+ 284us) client_negotiation.cc:240] Negotiation successful
0927 21:27:10.882050 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":282,"threads_started":1}
I20260927 21:27:10.884888 4848 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35003
I20260927 21:27:10.885229 4848 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:10.886005 4848 heartbeater.cc:511] Master 127.0.97.62:35003 requested a full tablet report, sending...
I20260927 21:27:10.887624 4527 ts_manager.cc:194] Registered new tserver with Master: 8f069afb2fa0402c97bf96231fee0e4d (127.0.97.2:33023)
I20260927 21:27:10.888902 4527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55215
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 21:27:10.917449 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35003
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:11.266019 4853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:11.266333 4853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:11.266518 4853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:11.274294 4853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:11.274439 4853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:11.274721 4853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:27:11.286103 4853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:11.287585 4853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:11.288985 4853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:11.303166 4858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:11.303208 4859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:11.303232 4861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:11.304574 4853 server_base.cc:1034] running on GCE node
I20260927 21:27:11.305346 4853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:11.306533 4853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:11.307819 4853 hybrid_clock.cc:648] HybridClock initialized: now 1790544431307726 us; error 91 us; skew 500 ppm
I20260927 21:27:11.308228 4853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:11 dist-test-slave-bxm0 krb5kdc[4483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:11.312049 4853 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 21:27:11.313375 4853 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:11.313529 4853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:11.313925 4853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:11.317804 4853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-2/data/instance:
uuid: "5249de7b9f4a4a7b9d2aa0aa729639d4"
format_stamp: "Formatted at 2026-09-27 21:27:11 on dist-test-slave-bxm0"
server_key: "b421a32457ca2afabd0bffb91f50525e"
server_key_iv: "644b61eac8eddf78f259be9c2411129c"
server_key_version: "encryptionkey@0"
I20260927 21:27:11.318804 4853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5249de7b9f4a4a7b9d2aa0aa729639d4"
format_stamp: "Formatted at 2026-09-27 21:27:11 on dist-test-slave-bxm0"
server_key: "b421a32457ca2afabd0bffb91f50525e"
server_key_iv: "644b61eac8eddf78f259be9c2411129c"
server_key_version: "encryptionkey@0"
I20260927 21:27:11.324759 4853 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 21:27:11.328684 4867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:11.330462 4853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:27:11.330675 4853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "5249de7b9f4a4a7b9d2aa0aa729639d4"
format_stamp: "Formatted at 2026-09-27 21:27:11 on dist-test-slave-bxm0"
server_key: "b421a32457ca2afabd0bffb91f50525e"
server_key_iv: "644b61eac8eddf78f259be9c2411129c"
server_key_version: "encryptionkey@0"
I20260927 21:27:11.330945 4853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:11.357245 4853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:11.362323 4853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:11.362654 4853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:11.363744 4853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:27:11.365255 4853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:11.365396 4853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:11.365548 4853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:11.365669 4853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:11.386538 4715 heartbeater.cc:503] Master 127.0.97.62:35003 was elected leader, sending a full tablet report...
I20260927 21:27:11.404107 4853 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39431
I20260927 21:27:11.404156 4979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39431 every 8 connection(s)
I20260927 21:27:11.405012 4853 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:27:11.408808 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 4853
I20260927 21:27:11.409010 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-2/wal/instance
I20260927 21:27:11.409540 388 external_mini_cluster.cc:1428] Setting key 9e0b890e7de000d09721d593357a7874
Sep 27 21:27:11 dist-test-slave-bxm0 krb5kdc[4483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:27:11.427205 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:11.407545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58469 (local address 127.0.97.62:35003)
0927 21:27:11.408165 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:11.408174 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:11.409342 (+ 1168us) server_negotiation.cc:407] Connection header received
0927 21:27:11.411669 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:11.411674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:11.411770 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:11.411891 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:11.413600 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.414414 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.416005 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.416457 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.416555 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:11.420553 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:11.420558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:11.420562 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:11.420599 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:11.423681 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:11.424823 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:11.424829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:11.424833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:11.425014 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:11.425550 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:11.425554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:11.425556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:11.425883 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:11.426012 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:11.426716 (+ 704us) server_negotiation.cc:299] Negotiation successful
0927 21:27:11.426924 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":320,"threads_started":1}
I20260927 21:27:11.428244 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:11.408189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35003 (local address 127.0.97.3:58469)
0927 21:27:11.409130 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:11.409178 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:11.411330 (+ 2152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:11.412164 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:11.412173 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:11.412833 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:11.413427 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:11.413446 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.414547 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.414550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:11.415831 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:11.415842 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.416702 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.416708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:11.416958 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:11.417833 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:11.417866 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:11.420293 (+ 2427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:11.423876 (+ 3583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:11.423949 (+ 73us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:11.423971 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:11.424672 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:11.425147 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:11.425152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:11.425156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:11.425434 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:11.426005 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:11.426011 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:11.426361 (+ 350us) client_negotiation.cc:769] Sending connection context
0927 21:27:11.426600 (+ 239us) client_negotiation.cc:240] Negotiation successful
0927 21:27:11.427032 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":351,"threads_started":1}
I20260927 21:27:11.429872 4980 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35003
I20260927 21:27:11.430209 4980 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:11.430827 4980 heartbeater.cc:511] Master 127.0.97.62:35003 requested a full tablet report, sending...
I20260927 21:27:11.432303 4527 ts_manager.cc:194] Registered new tserver with Master: 5249de7b9f4a4a7b9d2aa0aa729639d4 (127.0.97.3:39431)
I20260927 21:27:11.433087 4527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58469
I20260927 21:27:11.437978 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:27:11.456600 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:11.442696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46938 (local address 127.0.97.62:35003)
0927 21:27:11.442926 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:11.442932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:11.443341 (+ 409us) server_negotiation.cc:407] Connection header received
0927 21:27:11.443546 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:11.443553 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:11.443622 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:11.443732 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:11.445206 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.446031 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.447292 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.447733 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.447876 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:11.450561 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:11.450566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:11.450570 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:11.450608 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:11.453389 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:11.454225 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:11.454232 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:11.454236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:11.454393 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:11.455031 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:11.455037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:11.455041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:11.455404 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:11.455682 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:11.456019 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 21:27:11.456259 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 21:27:11.461148 4527 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:27:11.468205 4527 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:27:11.503530 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:11.480936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45982 (local address 127.0.97.2:33023)
0927 21:27:11.493096 (+ 12160us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:11.493111 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:11.493157 (+ 46us) server_negotiation.cc:407] Connection header received
0927 21:27:11.493590 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:11.493603 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:11.493983 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:11.494260 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:11.495588 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.497364 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.500472 (+ 3108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.502034 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.502167 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:11.502391 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:11.502933 (+ 542us) server_negotiation.cc:299] Negotiation successful
0927 21:27:11.503140 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11922,"thread_start_us":12359,"threads_started":1}
I20260927 21:27:11.504527 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:11.480746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33023 (local address 127.0.0.1:45982)
0927 21:27:11.491029 (+ 10283us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:11.491798 (+ 769us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:11.492089 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:11.494551 (+ 2462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:11.494565 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:11.494625 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:11.495139 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:11.495157 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.497570 (+ 2413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.497577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:11.500206 (+ 2629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:11.500225 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.502336 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.502343 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:11.502569 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:11.502635 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 21:27:11.502769 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 21:27:11.504164 (+ 1395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10002,"mutex_wait_us":1111,"thread_start_us":21201,"threads_started":1}
I20260927 21:27:11.507426 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:11.481443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50458 (local address 127.0.97.1:40071)
0927 21:27:11.496543 (+ 15100us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:11.496557 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:11.498029 (+ 1472us) server_negotiation.cc:407] Connection header received
0927 21:27:11.498136 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:11.498144 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:11.498431 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:11.498634 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:11.500646 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.502206 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.504795 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.506279 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.506433 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:11.506679 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:11.506946 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 21:27:11.507105 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14879,"thread_start_us":14711,"threads_started":1}
I20260927 21:27:11.508535 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:11.480886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40071 (local address 127.0.0.1:50458)
0927 21:27:11.497716 (+ 16830us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:11.497738 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:11.497881 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:11.499879 (+ 1998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:11.499884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:11.499918 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:11.500439 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:11.500450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.502400 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.502406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:11.504585 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:11.504600 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.506504 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.506511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:11.506721 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:11.506754 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 21:27:11.506895 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 21:27:11.507945 (+ 1050us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16650,"mutex_wait_us":33,"thread_start_us":9690,"threads_started":1}
I20260927 21:27:11.508776 4783 tablet_service.cc:1467] Processing CreateTablet for tablet b07c5b2e2ccd41fe969c133a847736db (DEFAULT_TABLE table=test-table [id=d8af94e2db754d509d995fd5366b2492]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:27:11.510433 4783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b07c5b2e2ccd41fe969c133a847736db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:11.510840 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:11.483353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39431 (local address 127.0.0.1:51144)
0927 21:27:11.502245 (+ 18892us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:11.502267 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:11.502420 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:11.503490 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:11.503497 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:11.503530 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:11.503937 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:11.503952 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.506062 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.506067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:11.508198 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:11.508213 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.510065 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.510072 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:11.510243 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:11.510278 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 21:27:11.510369 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 21:27:11.510516 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18757,"thread_start_us":15451,"threads_started":1}
I20260927 21:27:11.510897 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:11.483764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51144 (local address 127.0.97.3:39431)
0927 21:27:11.501485 (+ 17721us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:11.501498 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:11.502563 (+ 1065us) server_negotiation.cc:407] Connection header received
0927 21:27:11.502695 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:11.502706 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:11.502986 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:11.503186 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:11.504150 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.505901 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.508367 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.509856 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.510021 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:11.510254 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:11.510408 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 21:27:11.510572 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17467,"thread_start_us":17736,"threads_started":1}
I20260927 21:27:11.512585 4650 tablet_service.cc:1467] Processing CreateTablet for tablet b07c5b2e2ccd41fe969c133a847736db (DEFAULT_TABLE table=test-table [id=d8af94e2db754d509d995fd5366b2492]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:27:11.514081 4650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b07c5b2e2ccd41fe969c133a847736db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:11.516587 4915 tablet_service.cc:1467] Processing CreateTablet for tablet b07c5b2e2ccd41fe969c133a847736db (DEFAULT_TABLE table=test-table [id=d8af94e2db754d509d995fd5366b2492]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:27:11.517956 4915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b07c5b2e2ccd41fe969c133a847736db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:11.526661 4999 tablet_bootstrap.cc:492] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d: Bootstrap starting.
I20260927 21:27:11.528730 5000 tablet_bootstrap.cc:492] T b07c5b2e2ccd41fe969c133a847736db P 5ce2c7e47cd44c7096b5841c60f946e5: Bootstrap starting.
I20260927 21:27:11.530535 4999 tablet_bootstrap.cc:654] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:11.531675 5000 tablet_bootstrap.cc:654] T b07c5b2e2ccd41fe969c133a847736db P 5ce2c7e47cd44c7096b5841c60f946e5: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:11.532179 4999 log.cc:824] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d: Log is configured to *not* fsync() on all Append() calls
I20260927 21:27:11.532684 5000 log.cc:824] T b07c5b2e2ccd41fe969c133a847736db P 5ce2c7e47cd44c7096b5841c60f946e5: Log is configured to *not* fsync() on all Append() calls
I20260927 21:27:11.534313 5001 tablet_bootstrap.cc:492] T b07c5b2e2ccd41fe969c133a847736db P 5249de7b9f4a4a7b9d2aa0aa729639d4: Bootstrap starting.
I20260927 21:27:11.535841 5000 tablet_bootstrap.cc:492] T b07c5b2e2ccd41fe969c133a847736db P 5ce2c7e47cd44c7096b5841c60f946e5: No bootstrap required, opened a new log
I20260927 21:27:11.535841 4999 tablet_bootstrap.cc:492] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d: No bootstrap required, opened a new log
I20260927 21:27:11.536131 5000 ts_tablet_manager.cc:1397] T b07c5b2e2ccd41fe969c133a847736db P 5ce2c7e47cd44c7096b5841c60f946e5: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 21:27:11.536131 4999 ts_tablet_manager.cc:1397] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260927 21:27:11.538450 5001 tablet_bootstrap.cc:654] T b07c5b2e2ccd41fe969c133a847736db P 5249de7b9f4a4a7b9d2aa0aa729639d4: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:11.539548 5001 log.cc:824] T b07c5b2e2ccd41fe969c133a847736db P 5249de7b9f4a4a7b9d2aa0aa729639d4: Log is configured to *not* fsync() on all Append() calls
I20260927 21:27:11.542611 5001 tablet_bootstrap.cc:492] T b07c5b2e2ccd41fe969c133a847736db P 5249de7b9f4a4a7b9d2aa0aa729639d4: No bootstrap required, opened a new log
I20260927 21:27:11.542897 5001 ts_tablet_manager.cc:1397] T b07c5b2e2ccd41fe969c133a847736db P 5249de7b9f4a4a7b9d2aa0aa729639d4: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20260927 21:27:11.547145 5000 raft_consensus.cc:348] T b07c5b2e2ccd41fe969c133a847736db P 5ce2c7e47cd44c7096b5841c60f946e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f069afb2fa0402c97bf96231fee0e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } } peers { permanent_uuid: "5ce2c7e47cd44c7096b5841c60f946e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40071 } } peers { permanent_uuid: "5249de7b9f4a4a7b9d2aa0aa729639d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39431 } }
I20260927 21:27:11.547145 4999 raft_consensus.cc:348] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f069afb2fa0402c97bf96231fee0e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } } peers { permanent_uuid: "5ce2c7e47cd44c7096b5841c60f946e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40071 } } peers { permanent_uuid: "5249de7b9f4a4a7b9d2aa0aa729639d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39431 } }
I20260927 21:27:11.547627 4999 raft_consensus.cc:374] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:11.547627 5000 raft_consensus.cc:374] T b07c5b2e2ccd41fe969c133a847736db P 5ce2c7e47cd44c7096b5841c60f946e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:11.547734 5000 raft_consensus.cc:729] T b07c5b2e2ccd41fe969c133a847736db P 5ce2c7e47cd44c7096b5841c60f946e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ce2c7e47cd44c7096b5841c60f946e5, State: Initialized, Role: FOLLOWER
I20260927 21:27:11.547760 4999 raft_consensus.cc:729] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f069afb2fa0402c97bf96231fee0e4d, State: Initialized, Role: FOLLOWER
I20260927 21:27:11.548426 5000 consensus_queue.cc:260] T b07c5b2e2ccd41fe969c133a847736db P 5ce2c7e47cd44c7096b5841c60f946e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f069afb2fa0402c97bf96231fee0e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } } peers { permanent_uuid: "5ce2c7e47cd44c7096b5841c60f946e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40071 } } peers { permanent_uuid: "5249de7b9f4a4a7b9d2aa0aa729639d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39431 } }
I20260927 21:27:11.548422 4999 consensus_queue.cc:260] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f069afb2fa0402c97bf96231fee0e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } } peers { permanent_uuid: "5ce2c7e47cd44c7096b5841c60f946e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40071 } } peers { permanent_uuid: "5249de7b9f4a4a7b9d2aa0aa729639d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39431 } }
I20260927 21:27:11.550096 4848 heartbeater.cc:503] Master 127.0.97.62:35003 was elected leader, sending a full tablet report...
I20260927 21:27:11.550787 4999 ts_tablet_manager.cc:1428] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260927 21:27:11.550912 5000 ts_tablet_manager.cc:1428] T b07c5b2e2ccd41fe969c133a847736db P 5ce2c7e47cd44c7096b5841c60f946e5: Time spent starting tablet: real 0.015s user 0.008s sys 0.005s
I20260927 21:27:11.553153 5001 raft_consensus.cc:348] T b07c5b2e2ccd41fe969c133a847736db P 5249de7b9f4a4a7b9d2aa0aa729639d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f069afb2fa0402c97bf96231fee0e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } } peers { permanent_uuid: "5ce2c7e47cd44c7096b5841c60f946e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40071 } } peers { permanent_uuid: "5249de7b9f4a4a7b9d2aa0aa729639d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39431 } }
I20260927 21:27:11.553725 5001 raft_consensus.cc:374] T b07c5b2e2ccd41fe969c133a847736db P 5249de7b9f4a4a7b9d2aa0aa729639d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:11.553846 5001 raft_consensus.cc:729] T b07c5b2e2ccd41fe969c133a847736db P 5249de7b9f4a4a7b9d2aa0aa729639d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5249de7b9f4a4a7b9d2aa0aa729639d4, State: Initialized, Role: FOLLOWER
I20260927 21:27:11.554545 5001 consensus_queue.cc:260] T b07c5b2e2ccd41fe969c133a847736db P 5249de7b9f4a4a7b9d2aa0aa729639d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f069afb2fa0402c97bf96231fee0e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } } peers { permanent_uuid: "5ce2c7e47cd44c7096b5841c60f946e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40071 } } peers { permanent_uuid: "5249de7b9f4a4a7b9d2aa0aa729639d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39431 } }
I20260927 21:27:11.555743 4980 heartbeater.cc:503] Master 127.0.97.62:35003 was elected leader, sending a full tablet report...
I20260927 21:27:11.556377 5001 ts_tablet_manager.cc:1428] T b07c5b2e2ccd41fe969c133a847736db P 5249de7b9f4a4a7b9d2aa0aa729639d4: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
W20260927 21:27:11.602722 4716 tablet.cc:2369] T b07c5b2e2ccd41fe969c133a847736db P 5ce2c7e47cd44c7096b5841c60f946e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:27:11.613385 4849 tablet.cc:2369] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:27:11.650157 5006 raft_consensus.cc:482] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:27:11.650501 5006 raft_consensus.cc:504] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f069afb2fa0402c97bf96231fee0e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } } peers { permanent_uuid: "5ce2c7e47cd44c7096b5841c60f946e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40071 } } peers { permanent_uuid: "5249de7b9f4a4a7b9d2aa0aa729639d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39431 } }
I20260927 21:27:11.652370 5006 leader_election.cc:288] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ce2c7e47cd44c7096b5841c60f946e5 (127.0.97.1:40071), 5249de7b9f4a4a7b9d2aa0aa729639d4 (127.0.97.3:39431)
W20260927 21:27:11.656642 4981 tablet.cc:2369] T b07c5b2e2ccd41fe969c133a847736db P 5249de7b9f4a4a7b9d2aa0aa729639d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:27:11.659987 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:11.652857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32863 (local address 127.0.97.3:39431)
0927 21:27:11.653095 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:11.653104 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:11.654105 (+ 1001us) server_negotiation.cc:407] Connection header received
0927 21:27:11.654255 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:11.654260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:11.654332 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:11.654401 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:11.655056 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.655798 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.657941 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.659036 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.659197 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:11.659303 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:11.659586 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 21:27:11.659713 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 21:27:11.660152 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:11.652804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39431 (local address 127.0.97.2:32863)
0927 21:27:11.653998 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:11.654013 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:11.654124 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:11.654444 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:11.654451 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:11.654482 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:11.654897 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:11.654910 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.655940 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.655945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:11.657778 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:11.657791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.659262 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.659268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:11.659430 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:11.659453 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 21:27:11.659563 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 21:27:11.659708 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1042,"mutex_wait_us":53,"thread_start_us":593,"threads_started":1}
I20260927 21:27:11.660502 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:11.652812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34129 (local address 127.0.97.1:40071)
0927 21:27:11.653083 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:11.653093 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:11.654019 (+ 926us) server_negotiation.cc:407] Connection header received
0927 21:27:11.654321 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:11.654325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:11.654419 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:11.654511 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:11.655144 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.655826 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.658420 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.659535 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.659690 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:11.659801 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:11.660123 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 21:27:11.660250 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 21:27:11.660558 5008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:11.652613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40071 (local address 127.0.97.2:34129)
0927 21:27:11.653852 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:11.653872 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:11.654112 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:11.654548 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:11.654553 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:11.654591 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:11.654967 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:11.654975 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.656002 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.656005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:11.658239 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:11.658254 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.659761 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.659767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:11.659948 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:11.659970 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 21:27:11.660089 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 21:27:11.660227 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1127,"thread_start_us":715,"threads_started":1}
I20260927 21:27:11.661324 4935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b07c5b2e2ccd41fe969c133a847736db" candidate_uuid: "8f069afb2fa0402c97bf96231fee0e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5249de7b9f4a4a7b9d2aa0aa729639d4" is_pre_election: true
I20260927 21:27:11.661621 4670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b07c5b2e2ccd41fe969c133a847736db" candidate_uuid: "8f069afb2fa0402c97bf96231fee0e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ce2c7e47cd44c7096b5841c60f946e5" is_pre_election: true
I20260927 21:27:11.661836 4935 raft_consensus.cc:2454] T b07c5b2e2ccd41fe969c133a847736db P 5249de7b9f4a4a7b9d2aa0aa729639d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f069afb2fa0402c97bf96231fee0e4d in term 0.
I20260927 21:27:11.662161 4670 raft_consensus.cc:2454] T b07c5b2e2ccd41fe969c133a847736db P 5ce2c7e47cd44c7096b5841c60f946e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f069afb2fa0402c97bf96231fee0e4d in term 0.
I20260927 21:27:11.662550 4738 leader_election.cc:302] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5249de7b9f4a4a7b9d2aa0aa729639d4, 8f069afb2fa0402c97bf96231fee0e4d; no voters:
I20260927 21:27:11.663087 5006 raft_consensus.cc:2781] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:27:11.663242 5006 raft_consensus.cc:482] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:27:11.663365 5006 raft_consensus.cc:3037] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:11.665536 5006 raft_consensus.cc:504] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f069afb2fa0402c97bf96231fee0e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } } peers { permanent_uuid: "5ce2c7e47cd44c7096b5841c60f946e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40071 } } peers { permanent_uuid: "5249de7b9f4a4a7b9d2aa0aa729639d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39431 } }
I20260927 21:27:11.666332 5006 leader_election.cc:288] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d [CANDIDATE]: Term 1 election: Requested vote from peers 5ce2c7e47cd44c7096b5841c60f946e5 (127.0.97.1:40071), 5249de7b9f4a4a7b9d2aa0aa729639d4 (127.0.97.3:39431)
I20260927 21:27:11.666965 4670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b07c5b2e2ccd41fe969c133a847736db" candidate_uuid: "8f069afb2fa0402c97bf96231fee0e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ce2c7e47cd44c7096b5841c60f946e5"
I20260927 21:27:11.666985 4935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b07c5b2e2ccd41fe969c133a847736db" candidate_uuid: "8f069afb2fa0402c97bf96231fee0e4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5249de7b9f4a4a7b9d2aa0aa729639d4"
I20260927 21:27:11.667186 4670 raft_consensus.cc:3037] T b07c5b2e2ccd41fe969c133a847736db P 5ce2c7e47cd44c7096b5841c60f946e5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:11.667214 4935 raft_consensus.cc:3037] T b07c5b2e2ccd41fe969c133a847736db P 5249de7b9f4a4a7b9d2aa0aa729639d4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:11.669411 4670 raft_consensus.cc:2454] T b07c5b2e2ccd41fe969c133a847736db P 5ce2c7e47cd44c7096b5841c60f946e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f069afb2fa0402c97bf96231fee0e4d in term 1.
I20260927 21:27:11.669581 4935 raft_consensus.cc:2454] T b07c5b2e2ccd41fe969c133a847736db P 5249de7b9f4a4a7b9d2aa0aa729639d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f069afb2fa0402c97bf96231fee0e4d in term 1.
I20260927 21:27:11.670043 4737 leader_election.cc:302] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce2c7e47cd44c7096b5841c60f946e5, 8f069afb2fa0402c97bf96231fee0e4d; no voters:
I20260927 21:27:11.670425 5006 raft_consensus.cc:2781] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:27:11.670888 5006 raft_consensus.cc:686] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d [term 1 LEADER]: Becoming Leader. State: Replica: 8f069afb2fa0402c97bf96231fee0e4d, State: Running, Role: LEADER
I20260927 21:27:11.671525 5006 consensus_queue.cc:237] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f069afb2fa0402c97bf96231fee0e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } } peers { permanent_uuid: "5ce2c7e47cd44c7096b5841c60f946e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40071 } } peers { permanent_uuid: "5249de7b9f4a4a7b9d2aa0aa729639d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39431 } }
I20260927 21:27:11.676146 4527 catalog_manager.cc:5495] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f069afb2fa0402c97bf96231fee0e4d (127.0.97.2). New cstate: current_term: 1 leader_uuid: "8f069afb2fa0402c97bf96231fee0e4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f069afb2fa0402c97bf96231fee0e4d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ce2c7e47cd44c7096b5841c60f946e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5249de7b9f4a4a7b9d2aa0aa729639d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39431 } health_report { overall_health: UNKNOWN } } }
I20260927 21:27:11.700618 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:11.694883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45994 (local address 127.0.97.2:33023)
0927 21:27:11.695077 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:11.695081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:11.695201 (+ 120us) server_negotiation.cc:407] Connection header received
0927 21:27:11.695379 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:11.695382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:11.695463 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:11.695596 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:27:11.695972 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.696794 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.698404 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.698844 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.698974 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:11.699434 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:27:11.700114 (+ 680us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:27:11.700228 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:11.700322 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 21:27:11.700405 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 21:27:11.709364 4670 raft_consensus.cc:1261] T b07c5b2e2ccd41fe969c133a847736db P 5ce2c7e47cd44c7096b5841c60f946e5 [term 1 FOLLOWER]: Refusing update from remote peer 8f069afb2fa0402c97bf96231fee0e4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:27:11.709378 4935 raft_consensus.cc:1261] T b07c5b2e2ccd41fe969c133a847736db P 5249de7b9f4a4a7b9d2aa0aa729639d4 [term 1 FOLLOWER]: Refusing update from remote peer 8f069afb2fa0402c97bf96231fee0e4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:27:11.710542 5010 consensus_queue.cc:1035] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce2c7e47cd44c7096b5841c60f946e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:27:11.710891 5006 consensus_queue.cc:1035] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5249de7b9f4a4a7b9d2aa0aa729639d4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:27:11.724790 5013 mvcc.cc:205] Tried to move back new op lower bound from 7334069992270217216 to 7334069992130433024. Current Snapshot: MvccSnapshot[applied={T|T < 7334069992270217216}]
I20260927 21:27:11.738387 4527 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:27:11.738692 4527 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:27:11.744284 4527 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ad0c6c8444d4452972678ca04dabeb6: Sending DeleteTablet for 3 replicas of tablet b07c5b2e2ccd41fe969c133a847736db
I20260927 21:27:11.745886 4915 tablet_service.cc:1514] Processing DeleteTablet for tablet b07c5b2e2ccd41fe969c133a847736db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:27:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51144
I20260927 21:27:11.746610 4914 tablet_service.cc:1514] Processing DeleteTablet for tablet b07c5b2e2ccd41fe969c133a847736db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:27:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51144
I20260927 21:27:11.748023 4783 tablet_service.cc:1514] Processing DeleteTablet for tablet b07c5b2e2ccd41fe969c133a847736db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:27:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45982
I20260927 21:27:11.750175 4650 tablet_service.cc:1514] Processing DeleteTablet for tablet b07c5b2e2ccd41fe969c133a847736db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:27:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50458
I20260927 21:27:11.751183 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 4588
I20260927 21:27:11.757484 5022 tablet_replica.cc:330] T b07c5b2e2ccd41fe969c133a847736db P 5249de7b9f4a4a7b9d2aa0aa729639d4: stopping tablet replica
I20260927 21:27:11.758203 5022 raft_consensus.cc:2229] T b07c5b2e2ccd41fe969c133a847736db P 5249de7b9f4a4a7b9d2aa0aa729639d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:27:11.758855 5022 raft_consensus.cc:2258] T b07c5b2e2ccd41fe969c133a847736db P 5249de7b9f4a4a7b9d2aa0aa729639d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:27:11.761723 5022 ts_tablet_manager.cc:1905] T b07c5b2e2ccd41fe969c133a847736db P 5249de7b9f4a4a7b9d2aa0aa729639d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:27:11.765024 5024 tablet_replica.cc:330] T b07c5b2e2ccd41fe969c133a847736db P 5ce2c7e47cd44c7096b5841c60f946e5: stopping tablet replica
I20260927 21:27:11.765582 5024 raft_consensus.cc:2229] T b07c5b2e2ccd41fe969c133a847736db P 5ce2c7e47cd44c7096b5841c60f946e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:27:11.766193 5024 raft_consensus.cc:2258] T b07c5b2e2ccd41fe969c133a847736db P 5ce2c7e47cd44c7096b5841c60f946e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:27:11.766774 5023 tablet_replica.cc:330] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d: stopping tablet replica
I20260927 21:27:11.767261 5023 raft_consensus.cc:2229] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:27:11.767944 5023 raft_consensus.cc:2258] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:27:11.770260 5022 ts_tablet_manager.cc:1918] T b07c5b2e2ccd41fe969c133a847736db P 5249de7b9f4a4a7b9d2aa0aa729639d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:27:11.770539 5022 log.cc:1196] T b07c5b2e2ccd41fe969c133a847736db P 5249de7b9f4a4a7b9d2aa0aa729639d4: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-2/wal/wals/b07c5b2e2ccd41fe969c133a847736db
I20260927 21:27:11.770674 5023 ts_tablet_manager.cc:1905] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:27:11.771217 5022 ts_tablet_manager.cc:1939] T b07c5b2e2ccd41fe969c133a847736db P 5249de7b9f4a4a7b9d2aa0aa729639d4: Deleting consensus metadata
I20260927 21:27:11.773557 4513 catalog_manager.cc:4841] TS 5249de7b9f4a4a7b9d2aa0aa729639d4 (127.0.97.3:39431): tablet b07c5b2e2ccd41fe969c133a847736db (table test-table [id=d8af94e2db754d509d995fd5366b2492]) successfully deleted
I20260927 21:27:11.774467 5024 ts_tablet_manager.cc:1905] T b07c5b2e2ccd41fe969c133a847736db P 5ce2c7e47cd44c7096b5841c60f946e5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 21:27:11.774783 4513 catalog_manager.cc:4808] TS 5249de7b9f4a4a7b9d2aa0aa729639d4 (127.0.97.3:39431): delete failed for tablet b07c5b2e2ccd41fe969c133a847736db because the tablet was not found. No further retry: Not found: Tablet not found: b07c5b2e2ccd41fe969c133a847736db
I20260927 21:27:11.778360 5023 ts_tablet_manager.cc:1918] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:27:11.778674 5023 log.cc:1196] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-1/wal/wals/b07c5b2e2ccd41fe969c133a847736db
I20260927 21:27:11.779425 5023 ts_tablet_manager.cc:1939] T b07c5b2e2ccd41fe969c133a847736db P 8f069afb2fa0402c97bf96231fee0e4d: Deleting consensus metadata
Sep 27 21:27:11 dist-test-slave-bxm0 krb5kdc[4483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 21:27:11.781908 4511 catalog_manager.cc:4841] TS 8f069afb2fa0402c97bf96231fee0e4d (127.0.97.2:33023): tablet b07c5b2e2ccd41fe969c133a847736db (table test-table [id=d8af94e2db754d509d995fd5366b2492]) successfully deleted
I20260927 21:27:11.782883 5024 ts_tablet_manager.cc:1918] T b07c5b2e2ccd41fe969c133a847736db P 5ce2c7e47cd44c7096b5841c60f946e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:27:11.783087 5024 log.cc:1196] T b07c5b2e2ccd41fe969c133a847736db P 5ce2c7e47cd44c7096b5841c60f946e5: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790544390748963-388-0/minicluster-data/ts-0/wal/wals/b07c5b2e2ccd41fe969c133a847736db
I20260927 21:27:11.783735 5024 ts_tablet_manager.cc:1939] T b07c5b2e2ccd41fe969c133a847736db P 5ce2c7e47cd44c7096b5841c60f946e5: Deleting consensus metadata
I20260927 21:27:11.785753 4512 catalog_manager.cc:4841] TS 5ce2c7e47cd44c7096b5841c60f946e5 (127.0.97.1:40071): tablet b07c5b2e2ccd41fe969c133a847736db (table test-table [id=d8af94e2db754d509d995fd5366b2492]) successfully deleted
I20260927 21:27:11.789835 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:11.754007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50464 (local address 127.0.97.1:40071)
0927 21:27:11.754192 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:11.754201 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:11.762230 (+ 8029us) server_negotiation.cc:407] Connection header received
0927 21:27:11.762546 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:11.762552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:11.762689 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:11.762773 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:11.768414 (+ 5641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.769571 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.772110 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.772776 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.772883 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:11.782949 (+ 10066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:11.782963 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:11.782979 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:11.783077 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:11.786729 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:11.787568 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:11.787575 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:11.787578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:11.787719 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:11.788158 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:11.788162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:11.788165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:11.788528 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:11.788864 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:11.789184 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 21:27:11.789411 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 21:27:11.790612 4710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:11.880595 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 4588
I20260927 21:27:11.898690 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 4721
Sep 27 21:27:11 dist-test-slave-bxm0 krb5kdc[4483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 21:27:11.917301 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:11.899532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45998 (local address 127.0.97.2:33023)
0927 21:27:11.899691 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:11.899696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:11.899718 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:27:11.899925 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:11.899929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:11.900065 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:11.900151 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:11.901918 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.902695 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.904703 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:11.905158 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:11.905255 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:11.910765 (+ 5510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:11.910781 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:11.910797 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:11.910840 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:11.914307 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:11.915104 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:11.915112 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:11.915115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:11.915275 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:11.915676 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:11.915681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:11.915691 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:11.916052 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:11.916405 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:11.916707 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 21:27:11.916923 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:27:11.918100 4843 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:12.039016 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 4721
I20260927 21:27:12.058456 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 4853
Sep 27 21:27:12 dist-test-slave-bxm0 krb5kdc[4483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 21:27:12.074327 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:12.059245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51158 (local address 127.0.97.3:39431)
0927 21:27:12.059445 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:12.059454 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:12.059555 (+ 101us) server_negotiation.cc:407] Connection header received
0927 21:27:12.059683 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:12.059687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:12.059761 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:12.059848 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:12.061092 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:12.061883 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:12.063369 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:12.063788 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:12.063906 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:12.068324 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:12.068340 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:12.068357 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:12.068410 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:12.071352 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:12.072248 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:12.072261 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:12.072268 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:12.072508 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:12.072863 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:12.072866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:12.072869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:12.073249 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:12.073499 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:12.073766 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 21:27:12.073945 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 21:27:12.075165 4975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:12.165058 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 4853
I20260927 21:27:12.181803 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 4496
I20260927 21:27:12.183017 4557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:12.289850 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 4496
2026-09-27T21:27:12Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4332 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T21:27:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:27:12Z Disabled control of system clock
I20260927 21:27:12.357414 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36951
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 21:27:12.691306 5033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:12.691639 5033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:12.691790 5033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:12.699592 5033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:27:12.699728 5033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:12.699793 5033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:12.699877 5033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:27:12.699945 5033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:27:12.710096 5033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36951
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:12.711468 5033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:12.713330 5033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:12.726203 5041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:12.726334 5038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:12.726312 5039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:12.726854 5033 server_base.cc:1034] running on GCE node
I20260927 21:27:12.727790 5033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:12.729502 5033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:12.730748 5033 hybrid_clock.cc:648] HybridClock initialized: now 1790544432730672 us; error 79 us; skew 500 ppm
I20260927 21:27:12.731182 5033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:27:12.733875 5033 webserver.cc:460] Webserver started at http://127.0.97.62:35239/ using document root <none> and password file <none>
I20260927 21:27:12.734686 5033 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:12.734790 5033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:12.735220 5033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:12.738559 5033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "06cd7cfb85ed4b30b915b8db13e202ab"
format_stamp: "Formatted at 2026-09-27 21:27:12 on dist-test-slave-bxm0"
server_key: "d9e781c77b6fa926f469fffd9fb14fef"
server_key_iv: "407c2a55c8824e2627cfa3b40b74277f"
server_key_version: "encryptionkey@0"
I20260927 21:27:12.739470 5033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "06cd7cfb85ed4b30b915b8db13e202ab"
format_stamp: "Formatted at 2026-09-27 21:27:12 on dist-test-slave-bxm0"
server_key: "d9e781c77b6fa926f469fffd9fb14fef"
server_key_iv: "407c2a55c8824e2627cfa3b40b74277f"
server_key_version: "encryptionkey@0"
I20260927 21:27:12.745229 5033 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260927 21:27:12.749572 5047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:12.751178 5033 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 21:27:12.751430 5033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "06cd7cfb85ed4b30b915b8db13e202ab"
format_stamp: "Formatted at 2026-09-27 21:27:12 on dist-test-slave-bxm0"
server_key: "d9e781c77b6fa926f469fffd9fb14fef"
server_key_iv: "407c2a55c8824e2627cfa3b40b74277f"
server_key_version: "encryptionkey@0"
I20260927 21:27:12.751690 5033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:12.757997 5033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:12.758873 5033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:12.759202 5033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:12.778692 5033 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36951
I20260927 21:27:12.778728 5098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36951 every 8 connection(s)
I20260927 21:27:12.780200 5033 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:27:12.784366 5099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:12.788169 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 5033
I20260927 21:27:12.788395 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:27:12.789017 388 external_mini_cluster.cc:1428] Setting key f3cdabed5145830cde43d5d7b59b65c5
I20260927 21:27:12.796098 5099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06cd7cfb85ed4b30b915b8db13e202ab: Bootstrap starting.
I20260927 21:27:12.798604 5102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:12.791171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40528 (local address 127.0.97.62:36951)
0927 21:27:12.792259 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:12.792271 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:12.792314 (+ 43us) server_negotiation.cc:407] Connection header received
0927 21:27:12.794590 (+ 2276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:12.794619 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:12.795129 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:12.795554 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:12.796805 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:12.796811 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:12.796824 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:27:12.796837 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:12.796893 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:27:12.796911 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:12.797163 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:12.797301 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 21:27:12.797568 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":376,"threads_started":1}
I20260927 21:27:12.799860 5099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06cd7cfb85ed4b30b915b8db13e202ab: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:12.801301 5099 log.cc:824] T 00000000000000000000000000000000 P 06cd7cfb85ed4b30b915b8db13e202ab: Log is configured to *not* fsync() on all Append() calls
I20260927 21:27:12.804246 5099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06cd7cfb85ed4b30b915b8db13e202ab: No bootstrap required, opened a new log
I20260927 21:27:12.812055 5099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06cd7cfb85ed4b30b915b8db13e202ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cd7cfb85ed4b30b915b8db13e202ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36951 } }
I20260927 21:27:12.812439 5099 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06cd7cfb85ed4b30b915b8db13e202ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:12.812520 5099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06cd7cfb85ed4b30b915b8db13e202ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06cd7cfb85ed4b30b915b8db13e202ab, State: Initialized, Role: FOLLOWER
I20260927 21:27:12.813105 5099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06cd7cfb85ed4b30b915b8db13e202ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cd7cfb85ed4b30b915b8db13e202ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36951 } }
I20260927 21:27:12.813314 5099 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06cd7cfb85ed4b30b915b8db13e202ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:27:12.813405 5099 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06cd7cfb85ed4b30b915b8db13e202ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:27:12.813534 5099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06cd7cfb85ed4b30b915b8db13e202ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:12.816254 5099 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06cd7cfb85ed4b30b915b8db13e202ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cd7cfb85ed4b30b915b8db13e202ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36951 } }
I20260927 21:27:12.816751 5099 leader_election.cc:302] T 00000000000000000000000000000000 P 06cd7cfb85ed4b30b915b8db13e202ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06cd7cfb85ed4b30b915b8db13e202ab; no voters:
I20260927 21:27:12.817286 5099 leader_election.cc:288] T 00000000000000000000000000000000 P 06cd7cfb85ed4b30b915b8db13e202ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:27:12.817476 5104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06cd7cfb85ed4b30b915b8db13e202ab [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:27:12.818245 5104 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06cd7cfb85ed4b30b915b8db13e202ab [term 1 LEADER]: Becoming Leader. State: Replica: 06cd7cfb85ed4b30b915b8db13e202ab, State: Running, Role: LEADER
I20260927 21:27:12.818811 5104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06cd7cfb85ed4b30b915b8db13e202ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cd7cfb85ed4b30b915b8db13e202ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36951 } }
I20260927 21:27:12.819505 5099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06cd7cfb85ed4b30b915b8db13e202ab [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:27:12.821853 5106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06cd7cfb85ed4b30b915b8db13e202ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06cd7cfb85ed4b30b915b8db13e202ab. Latest consensus state: current_term: 1 leader_uuid: "06cd7cfb85ed4b30b915b8db13e202ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cd7cfb85ed4b30b915b8db13e202ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36951 } } }
I20260927 21:27:12.821914 5105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06cd7cfb85ed4b30b915b8db13e202ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06cd7cfb85ed4b30b915b8db13e202ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06cd7cfb85ed4b30b915b8db13e202ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36951 } } }
I20260927 21:27:12.822268 5106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06cd7cfb85ed4b30b915b8db13e202ab [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:12.822269 5105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06cd7cfb85ed4b30b915b8db13e202ab [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:12.824270 5110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:27:12.829546 5110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:27:12.836314 5121 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 21:27:12.838572 5110 catalog_manager.cc:1347] Generated new cluster ID: e3e6a97dcbf34a3d9861bb1a9054a80d
I20260927 21:27:12.838686 5110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:27:12.860837 5110 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:27:12.861370 5110 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:27:12.867378 5110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06cd7cfb85ed4b30b915b8db13e202ab: Generated new TSK 0
I20260927 21:27:12.868235 5110 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:27:12.886977 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36951
--builtin_ntp_servers=127.0.97.20:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 21:27:13.240789 5123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:13.241118 5123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:13.241267 5123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:13.249089 5123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:13.249208 5123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:13.249402 5123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:27:13.259526 5123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:13.260906 5123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:13.263062 5123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:13.276713 5128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:13.276820 5129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:13.276841 5131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:13.278040 5123 server_base.cc:1034] running on GCE node
I20260927 21:27:13.278718 5123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:13.279810 5123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:13.281167 5123 hybrid_clock.cc:648] HybridClock initialized: now 1790544433281098 us; error 69 us; skew 500 ppm
I20260927 21:27:13.281538 5123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:27:13.284140 5123 webserver.cc:460] Webserver started at http://127.0.97.1:43667/ using document root <none> and password file <none>
I20260927 21:27:13.284962 5123 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:13.285116 5123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:13.285463 5123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:13.288698 5123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-0/data/instance:
uuid: "a68bc17903cb4f7ca236d4e9bc103a0f"
format_stamp: "Formatted at 2026-09-27 21:27:13 on dist-test-slave-bxm0"
server_key: "780228d1705303fd7b7148b8c20b57cf"
server_key_iv: "9fc4a8b5d0c16f2677e85f1c3d844dae"
server_key_version: "encryptionkey@0"
I20260927 21:27:13.289594 5123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a68bc17903cb4f7ca236d4e9bc103a0f"
format_stamp: "Formatted at 2026-09-27 21:27:13 on dist-test-slave-bxm0"
server_key: "780228d1705303fd7b7148b8c20b57cf"
server_key_iv: "9fc4a8b5d0c16f2677e85f1c3d844dae"
server_key_version: "encryptionkey@0"
I20260927 21:27:13.295909 5123 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 21:27:13.300102 5137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:13.301867 5123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 21:27:13.302096 5123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "a68bc17903cb4f7ca236d4e9bc103a0f"
format_stamp: "Formatted at 2026-09-27 21:27:13 on dist-test-slave-bxm0"
server_key: "780228d1705303fd7b7148b8c20b57cf"
server_key_iv: "9fc4a8b5d0c16f2677e85f1c3d844dae"
server_key_version: "encryptionkey@0"
I20260927 21:27:13.302327 5123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:13.308720 5123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:13.309715 5123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:13.310009 5123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:13.311100 5123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:27:13.312670 5123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:13.312762 5123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:13.312935 5123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:13.313043 5123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:13.350152 5123 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35675
I20260927 21:27:13.350217 5249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35675 every 8 connection(s)
I20260927 21:27:13.351604 5123 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:27:13.358642 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 5123
I20260927 21:27:13.358871 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-0/wal/instance
I20260927 21:27:13.359534 388 external_mini_cluster.cc:1428] Setting key 522802fb5a7929d7515b6292e8217de5
I20260927 21:27:13.360613 5252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:13.354401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52117 (local address 127.0.97.62:36951)
0927 21:27:13.354874 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:13.354882 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:13.356100 (+ 1218us) server_negotiation.cc:407] Connection header received
0927 21:27:13.357988 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:13.357995 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:13.358070 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:13.358224 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:13.359800 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:13.359807 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:13.359811 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:27:13.359836 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:13.359900 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:27:13.359919 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:13.360071 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:13.360259 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 21:27:13.360315 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":258,"threads_started":1}
I20260927 21:27:13.361835 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:13.354862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36951 (local address 127.0.97.1:52117)
0927 21:27:13.355913 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:13.355958 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:13.357674 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:13.358502 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:13.358511 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:13.358622 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:13.359425 (+ 803us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:27:13.359427 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:13.359471 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:27:13.359567 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:13.360043 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:13.360050 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:13.360056 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 21:27:13.360324 (+ 268us) client_negotiation.cc:240] Negotiation successful
0927 21:27:13.360624 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":292,"threads_started":1}
I20260927 21:27:13.362722 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36951
--builtin_ntp_servers=127.0.97.20:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 21:27:13.363598 5250 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36951
I20260927 21:27:13.363929 5250 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:13.364548 5250 heartbeater.cc:511] Master 127.0.97.62:36951 requested a full tablet report, sending...
I20260927 21:27:13.366890 5064 ts_manager.cc:194] Registered new tserver with Master: a68bc17903cb4f7ca236d4e9bc103a0f (127.0.97.1:35675)
W20260927 21:27:13.712850 5254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:13.713167 5254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:13.713258 5254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:13.720767 5254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:13.720904 5254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:13.721040 5254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:27:13.730789 5254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:13.732084 5254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:13.733906 5254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:13.747272 5259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:13.747326 5260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:13.747359 5262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:13.748237 5254 server_base.cc:1034] running on GCE node
I20260927 21:27:13.748909 5254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:13.750029 5254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:13.751232 5254 hybrid_clock.cc:648] HybridClock initialized: now 1790544433751185 us; error 37 us; skew 500 ppm
I20260927 21:27:13.751577 5254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:27:13.754583 5254 webserver.cc:460] Webserver started at http://127.0.97.2:37345/ using document root <none> and password file <none>
I20260927 21:27:13.755391 5254 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:13.755544 5254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:13.755966 5254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:13.759325 5254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-1/data/instance:
uuid: "39c15b04b7164fb39bb8f2c795b1ef0e"
format_stamp: "Formatted at 2026-09-27 21:27:13 on dist-test-slave-bxm0"
server_key: "57a731d89a99436068096681e2b66c6e"
server_key_iv: "1fafd93efabe882a06b1fe15ef5aee4c"
server_key_version: "encryptionkey@0"
I20260927 21:27:13.760214 5254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-1/wal/instance:
uuid: "39c15b04b7164fb39bb8f2c795b1ef0e"
format_stamp: "Formatted at 2026-09-27 21:27:13 on dist-test-slave-bxm0"
server_key: "57a731d89a99436068096681e2b66c6e"
server_key_iv: "1fafd93efabe882a06b1fe15ef5aee4c"
server_key_version: "encryptionkey@0"
I20260927 21:27:13.766383 5254 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 21:27:13.770694 5268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:13.772217 5254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260927 21:27:13.772495 5254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "39c15b04b7164fb39bb8f2c795b1ef0e"
format_stamp: "Formatted at 2026-09-27 21:27:13 on dist-test-slave-bxm0"
server_key: "57a731d89a99436068096681e2b66c6e"
server_key_iv: "1fafd93efabe882a06b1fe15ef5aee4c"
server_key_version: "encryptionkey@0"
I20260927 21:27:13.772770 5254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:13.779505 5254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:13.780550 5254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:13.780897 5254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:13.782027 5254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:27:13.783845 5254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:13.783952 5254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:13.784060 5254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:13.784130 5254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:13.820922 5254 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38511
I20260927 21:27:13.820978 5380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38511 every 8 connection(s)
I20260927 21:27:13.822414 5254 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:27:13.830124 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 5254
I20260927 21:27:13.830299 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-1/wal/instance
I20260927 21:27:13.830713 5252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:13.825056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50509 (local address 127.0.97.62:36951)
0927 21:27:13.825256 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:13.825260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:13.826696 (+ 1436us) server_negotiation.cc:407] Connection header received
0927 21:27:13.828358 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:13.828364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:13.828436 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:13.828561 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:13.829921 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:13.829928 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:13.829934 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:27:13.829955 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:13.829984 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:27:13.829995 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:13.830122 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:13.830375 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 21:27:13.830445 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 21:27:13.830770 388 external_mini_cluster.cc:1428] Setting key 7d8d1bf2b0b3694a42234cabc89c4644
I20260927 21:27:13.831861 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:13.825585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36951 (local address 127.0.97.2:50509)
0927 21:27:13.826533 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:13.826575 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:13.828090 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:13.828787 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:13.828795 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:13.828891 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:13.829550 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:27:13.829553 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:13.829590 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:27:13.829738 (+ 148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:13.830089 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:13.830099 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:13.830106 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 21:27:13.830476 (+ 370us) client_negotiation.cc:240] Negotiation successful
0927 21:27:13.830744 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":280,"threads_started":1}
I20260927 21:27:13.833338 5381 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36951
I20260927 21:27:13.833725 5381 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:13.833806 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36951
--builtin_ntp_servers=127.0.97.20:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 21:27:13.834434 5381 heartbeater.cc:511] Master 127.0.97.62:36951 requested a full tablet report, sending...
I20260927 21:27:13.836040 5064 ts_manager.cc:194] Registered new tserver with Master: 39c15b04b7164fb39bb8f2c795b1ef0e (127.0.97.2:38511)
W20260927 21:27:14.172608 5384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:14.172968 5384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:14.173143 5384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:14.180850 5384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:14.181037 5384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:14.181241 5384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:27:14.190737 5384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:14.192127 5384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:14.194080 5384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:14.208261 5389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:14.208339 5392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:14.208406 5390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:14.209477 5384 server_base.cc:1034] running on GCE node
I20260927 21:27:14.210204 5384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:14.211252 5384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:14.212495 5384 hybrid_clock.cc:648] HybridClock initialized: now 1790544434212437 us; error 58 us; skew 500 ppm
I20260927 21:27:14.212945 5384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:27:14.215595 5384 webserver.cc:460] Webserver started at http://127.0.97.3:41385/ using document root <none> and password file <none>
I20260927 21:27:14.216408 5384 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:14.216562 5384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:14.216902 5384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:14.220171 5384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-2/data/instance:
uuid: "7a96b7178c3a410c937d021de071a860"
format_stamp: "Formatted at 2026-09-27 21:27:14 on dist-test-slave-bxm0"
server_key: "d6c45f44e5bfa5b0958794a80db8dd13"
server_key_iv: "4635c8c0ff7cf3b2624b777a05f7eab4"
server_key_version: "encryptionkey@0"
I20260927 21:27:14.221397 5384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7a96b7178c3a410c937d021de071a860"
format_stamp: "Formatted at 2026-09-27 21:27:14 on dist-test-slave-bxm0"
server_key: "d6c45f44e5bfa5b0958794a80db8dd13"
server_key_iv: "4635c8c0ff7cf3b2624b777a05f7eab4"
server_key_version: "encryptionkey@0"
I20260927 21:27:14.227291 5384 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 21:27:14.231045 5398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:14.232628 5384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:27:14.232798 5384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "7a96b7178c3a410c937d021de071a860"
format_stamp: "Formatted at 2026-09-27 21:27:14 on dist-test-slave-bxm0"
server_key: "d6c45f44e5bfa5b0958794a80db8dd13"
server_key_iv: "4635c8c0ff7cf3b2624b777a05f7eab4"
server_key_version: "encryptionkey@0"
I20260927 21:27:14.233060 5384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:14.239202 5384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:14.240042 5384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:14.240305 5384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:14.241222 5384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:27:14.242789 5384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:14.242880 5384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:14.242978 5384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:14.243043 5384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:14.277107 5384 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43469
I20260927 21:27:14.277206 5510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43469 every 8 connection(s)
I20260927 21:27:14.278712 5384 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:27:14.284329 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 5384
I20260927 21:27:14.284474 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-2/wal/instance
I20260927 21:27:14.285121 388 external_mini_cluster.cc:1428] Setting key fcee756ecf958f9abfadbe822792f739
I20260927 21:27:14.288669 5252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:14.281443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38477 (local address 127.0.97.62:36951)
0927 21:27:14.281699 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:14.281706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:14.283502 (+ 1796us) server_negotiation.cc:407] Connection header received
0927 21:27:14.285710 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:14.285719 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:14.285807 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:14.285915 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:14.287673 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:14.287682 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:14.287687 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:27:14.287711 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:14.287760 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:27:14.287780 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:14.287922 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:14.288212 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 21:27:14.288283 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 21:27:14.289981 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:14.282144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36951 (local address 127.0.97.3:38477)
0927 21:27:14.283283 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:14.283329 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:14.285265 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:14.286237 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:14.286247 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:14.286366 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:14.287276 (+ 910us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:27:14.287280 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:14.287340 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:27:14.287483 (+ 143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:14.287953 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:14.287962 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:14.287969 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 21:27:14.288245 (+ 276us) client_negotiation.cc:240] Negotiation successful
0927 21:27:14.288582 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":347,"threads_started":1}
I20260927 21:27:14.291479 5511 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36951
I20260927 21:27:14.291760 5511 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:14.292336 5511 heartbeater.cc:511] Master 127.0.97.62:36951 requested a full tablet report, sending...
I20260927 21:27:14.294010 5064 ts_manager.cc:194] Registered new tserver with Master: 7a96b7178c3a410c937d021de071a860 (127.0.97.3:43469)
I20260927 21:27:14.302645 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:27:14.310179 5252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:14.307597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40544 (local address 127.0.97.62:36951)
0927 21:27:14.307833 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:14.307841 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:14.308215 (+ 374us) server_negotiation.cc:407] Connection header received
0927 21:27:14.308343 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:14.308347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:14.308407 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:14.308495 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:14.309419 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:14.309423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:14.309426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:27:14.309443 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:14.309475 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:27:14.309487 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:14.309596 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:14.309810 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 21:27:14.309865 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 21:27:14.314498 5064 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:27:14.321593 5064 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:27:14.368163 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:14.335521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52326 (local address 127.0.97.3:43469)
0927 21:27:14.364644 (+ 29123us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:14.364662 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:14.364690 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:27:14.364748 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:14.364754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:14.364967 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:14.365794 (+ 827us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:14.367197 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:14.367202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:14.367212 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:27:14.367225 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:14.367259 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:27:14.367273 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:14.367461 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:14.367602 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 21:27:14.367716 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28952,"thread_start_us":28780,"threads_started":1}
I20260927 21:27:14.368296 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:14.335055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43469 (local address 127.0.0.1:52326)
0927 21:27:14.346556 (+ 11501us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:14.346612 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:14.346795 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:14.365178 (+ 18383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:14.365183 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:14.365226 (+ 43us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:14.365952 (+ 726us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:27:14.365955 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:14.365972 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:27:14.366020 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:14.367380 (+ 1360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:14.367391 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:14.367410 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 21:27:14.367631 (+ 221us) client_negotiation.cc:240] Negotiation successful
0927 21:27:14.367742 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11203,"thread_start_us":26005,"threads_started":1}
I20260927 21:27:14.368367 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:14.335135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35675 (local address 127.0.0.1:49820)
0927 21:27:14.352060 (+ 16925us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:14.352087 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:14.352224 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:14.364257 (+ 12033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:14.364269 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:14.364386 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:14.365544 (+ 1158us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:27:14.365547 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:14.365580 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:27:14.365963 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:14.367522 (+ 1559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:14.367528 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:14.367531 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:27:14.367735 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 21:27:14.367798 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16698,"thread_start_us":10950,"threads_started":1}
I20260927 21:27:14.368433 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:14.335532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38511 (local address 127.0.0.1:53796)
0927 21:27:14.364248 (+ 28716us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:14.364271 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:14.364420 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:14.366175 (+ 1755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:14.366179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:14.366201 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:14.366891 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:27:14.366894 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:14.366911 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:27:14.366954 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:14.367670 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:14.367677 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:14.367689 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 21:27:14.367832 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 21:27:14.367877 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28557,"thread_start_us":28406,"threads_started":1}
I20260927 21:27:14.368999 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:14.336426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53796 (local address 127.0.97.2:38511)
0927 21:27:14.363241 (+ 26815us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:14.363256 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:14.364577 (+ 1321us) server_negotiation.cc:407] Connection header received
0927 21:27:14.364659 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:14.364667 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:14.364894 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:14.365082 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:14.367134 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:14.367143 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:14.367159 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:27:14.367188 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:14.367243 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:27:14.367264 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:14.367442 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:14.368590 (+ 1148us) server_negotiation.cc:299] Negotiation successful
0927 21:27:14.368692 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26547,"thread_start_us":26836,"threads_started":1}
I20260927 21:27:14.369503 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:14.335983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49820 (local address 127.0.97.1:35675)
0927 21:27:14.363373 (+ 27390us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:14.363387 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:14.363429 (+ 42us) server_negotiation.cc:407] Connection header received
0927 21:27:14.363527 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:14.363540 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:14.363895 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:14.366153 (+ 2258us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:14.367156 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:14.367162 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:14.367173 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:27:14.367188 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:14.367243 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:27:14.367263 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:14.367456 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:14.368613 (+ 1157us) server_negotiation.cc:299] Negotiation successful
0927 21:27:14.368692 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27143,"thread_start_us":27531,"threads_started":1}
I20260927 21:27:14.370870 5250 heartbeater.cc:503] Master 127.0.97.62:36951 was elected leader, sending a full tablet report...
I20260927 21:27:14.372977 5316 tablet_service.cc:1467] Processing CreateTablet for tablet 8ef60bc6e2f041d58d919b87fdc44634 (DEFAULT_TABLE table=test-table [id=7c4c816ecd574a71a99ba7875947eab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:27:14.372963 5446 tablet_service.cc:1467] Processing CreateTablet for tablet 8ef60bc6e2f041d58d919b87fdc44634 (DEFAULT_TABLE table=test-table [id=7c4c816ecd574a71a99ba7875947eab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:27:14.372962 5185 tablet_service.cc:1467] Processing CreateTablet for tablet 8ef60bc6e2f041d58d919b87fdc44634 (DEFAULT_TABLE table=test-table [id=7c4c816ecd574a71a99ba7875947eab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:27:14.374277 5316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ef60bc6e2f041d58d919b87fdc44634. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:14.374351 5185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ef60bc6e2f041d58d919b87fdc44634. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:14.374367 5446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ef60bc6e2f041d58d919b87fdc44634. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:14.386842 5529 tablet_bootstrap.cc:492] T 8ef60bc6e2f041d58d919b87fdc44634 P 39c15b04b7164fb39bb8f2c795b1ef0e: Bootstrap starting.
I20260927 21:27:14.390367 5530 tablet_bootstrap.cc:492] T 8ef60bc6e2f041d58d919b87fdc44634 P a68bc17903cb4f7ca236d4e9bc103a0f: Bootstrap starting.
I20260927 21:27:14.391957 5529 tablet_bootstrap.cc:654] T 8ef60bc6e2f041d58d919b87fdc44634 P 39c15b04b7164fb39bb8f2c795b1ef0e: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:14.392383 5531 tablet_bootstrap.cc:492] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860: Bootstrap starting.
I20260927 21:27:14.393594 5529 log.cc:824] T 8ef60bc6e2f041d58d919b87fdc44634 P 39c15b04b7164fb39bb8f2c795b1ef0e: Log is configured to *not* fsync() on all Append() calls
I20260927 21:27:14.394306 5530 tablet_bootstrap.cc:654] T 8ef60bc6e2f041d58d919b87fdc44634 P a68bc17903cb4f7ca236d4e9bc103a0f: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:14.395614 5530 log.cc:824] T 8ef60bc6e2f041d58d919b87fdc44634 P a68bc17903cb4f7ca236d4e9bc103a0f: Log is configured to *not* fsync() on all Append() calls
I20260927 21:27:14.396134 5531 tablet_bootstrap.cc:654] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:14.397128 5531 log.cc:824] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860: Log is configured to *not* fsync() on all Append() calls
I20260927 21:27:14.398195 5529 tablet_bootstrap.cc:492] T 8ef60bc6e2f041d58d919b87fdc44634 P 39c15b04b7164fb39bb8f2c795b1ef0e: No bootstrap required, opened a new log
I20260927 21:27:14.398506 5529 ts_tablet_manager.cc:1397] T 8ef60bc6e2f041d58d919b87fdc44634 P 39c15b04b7164fb39bb8f2c795b1ef0e: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260927 21:27:14.400115 5530 tablet_bootstrap.cc:492] T 8ef60bc6e2f041d58d919b87fdc44634 P a68bc17903cb4f7ca236d4e9bc103a0f: No bootstrap required, opened a new log
I20260927 21:27:14.400405 5531 tablet_bootstrap.cc:492] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860: No bootstrap required, opened a new log
I20260927 21:27:14.400410 5530 ts_tablet_manager.cc:1397] T 8ef60bc6e2f041d58d919b87fdc44634 P a68bc17903cb4f7ca236d4e9bc103a0f: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 21:27:14.400665 5531 ts_tablet_manager.cc:1397] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20260927 21:27:14.408968 5530 raft_consensus.cc:348] T 8ef60bc6e2f041d58d919b87fdc44634 P a68bc17903cb4f7ca236d4e9bc103a0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a96b7178c3a410c937d021de071a860" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43469 } } peers { permanent_uuid: "a68bc17903cb4f7ca236d4e9bc103a0f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35675 } } peers { permanent_uuid: "39c15b04b7164fb39bb8f2c795b1ef0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38511 } }
I20260927 21:27:14.409415 5530 raft_consensus.cc:374] T 8ef60bc6e2f041d58d919b87fdc44634 P a68bc17903cb4f7ca236d4e9bc103a0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:14.409515 5530 raft_consensus.cc:729] T 8ef60bc6e2f041d58d919b87fdc44634 P a68bc17903cb4f7ca236d4e9bc103a0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a68bc17903cb4f7ca236d4e9bc103a0f, State: Initialized, Role: FOLLOWER
I20260927 21:27:14.410228 5530 consensus_queue.cc:260] T 8ef60bc6e2f041d58d919b87fdc44634 P a68bc17903cb4f7ca236d4e9bc103a0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a96b7178c3a410c937d021de071a860" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43469 } } peers { permanent_uuid: "a68bc17903cb4f7ca236d4e9bc103a0f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35675 } } peers { permanent_uuid: "39c15b04b7164fb39bb8f2c795b1ef0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38511 } }
I20260927 21:27:14.410152 5529 raft_consensus.cc:348] T 8ef60bc6e2f041d58d919b87fdc44634 P 39c15b04b7164fb39bb8f2c795b1ef0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a96b7178c3a410c937d021de071a860" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43469 } } peers { permanent_uuid: "a68bc17903cb4f7ca236d4e9bc103a0f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35675 } } peers { permanent_uuid: "39c15b04b7164fb39bb8f2c795b1ef0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38511 } }
I20260927 21:27:14.410635 5529 raft_consensus.cc:374] T 8ef60bc6e2f041d58d919b87fdc44634 P 39c15b04b7164fb39bb8f2c795b1ef0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:14.410748 5529 raft_consensus.cc:729] T 8ef60bc6e2f041d58d919b87fdc44634 P 39c15b04b7164fb39bb8f2c795b1ef0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c15b04b7164fb39bb8f2c795b1ef0e, State: Initialized, Role: FOLLOWER
I20260927 21:27:14.411464 5529 consensus_queue.cc:260] T 8ef60bc6e2f041d58d919b87fdc44634 P 39c15b04b7164fb39bb8f2c795b1ef0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a96b7178c3a410c937d021de071a860" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43469 } } peers { permanent_uuid: "a68bc17903cb4f7ca236d4e9bc103a0f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35675 } } peers { permanent_uuid: "39c15b04b7164fb39bb8f2c795b1ef0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38511 } }
I20260927 21:27:14.411597 5531 raft_consensus.cc:348] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a96b7178c3a410c937d021de071a860" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43469 } } peers { permanent_uuid: "a68bc17903cb4f7ca236d4e9bc103a0f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35675 } } peers { permanent_uuid: "39c15b04b7164fb39bb8f2c795b1ef0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38511 } }
I20260927 21:27:14.412063 5531 raft_consensus.cc:374] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:14.412181 5531 raft_consensus.cc:729] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a96b7178c3a410c937d021de071a860, State: Initialized, Role: FOLLOWER
I20260927 21:27:14.412251 5530 ts_tablet_manager.cc:1428] T 8ef60bc6e2f041d58d919b87fdc44634 P a68bc17903cb4f7ca236d4e9bc103a0f: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260927 21:27:14.412966 5531 consensus_queue.cc:260] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a96b7178c3a410c937d021de071a860" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43469 } } peers { permanent_uuid: "a68bc17903cb4f7ca236d4e9bc103a0f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35675 } } peers { permanent_uuid: "39c15b04b7164fb39bb8f2c795b1ef0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38511 } }
I20260927 21:27:14.413189 5381 heartbeater.cc:503] Master 127.0.97.62:36951 was elected leader, sending a full tablet report...
I20260927 21:27:14.414067 5529 ts_tablet_manager.cc:1428] T 8ef60bc6e2f041d58d919b87fdc44634 P 39c15b04b7164fb39bb8f2c795b1ef0e: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260927 21:27:14.415337 5511 heartbeater.cc:503] Master 127.0.97.62:36951 was elected leader, sending a full tablet report...
I20260927 21:27:14.416079 5531 ts_tablet_manager.cc:1428] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860: Time spent starting tablet: real 0.015s user 0.008s sys 0.009s
I20260927 21:27:14.505760 5537 raft_consensus.cc:482] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:27:14.506129 5537 raft_consensus.cc:504] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a96b7178c3a410c937d021de071a860" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43469 } } peers { permanent_uuid: "a68bc17903cb4f7ca236d4e9bc103a0f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35675 } } peers { permanent_uuid: "39c15b04b7164fb39bb8f2c795b1ef0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38511 } }
I20260927 21:27:14.508008 5537 leader_election.cc:288] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a68bc17903cb4f7ca236d4e9bc103a0f (127.0.97.1:35675), 39c15b04b7164fb39bb8f2c795b1ef0e (127.0.97.2:38511)
I20260927 21:27:14.511111 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:14.508517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46667 (local address 127.0.97.1:35675)
0927 21:27:14.508706 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:14.508715 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:14.508827 (+ 112us) server_negotiation.cc:407] Connection header received
0927 21:27:14.508992 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:14.509000 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:14.509071 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:14.509215 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:14.510502 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:14.510507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:14.510510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:27:14.510525 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:14.510554 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:27:14.510566 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:14.510693 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:14.510784 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 21:27:14.510849 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 21:27:14.511164 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:14.508491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35675 (local address 127.0.97.3:46667)
0927 21:27:14.508704 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:14.508725 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:14.508860 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:14.509302 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:14.509311 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:14.509364 (+ 53us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:14.510238 (+ 874us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:27:14.510241 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:14.510261 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:27:14.510313 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:14.510664 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:14.510670 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:14.510675 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:27:14.510782 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 21:27:14.510861 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 21:27:14.512019 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:14.508895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38511 (local address 127.0.97.3:40327)
0927 21:27:14.509695 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:14.509718 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:14.509879 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:14.510234 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:14.510239 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:14.510262 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:14.510972 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:27:14.510976 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:14.510992 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:27:14.511033 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:14.511371 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:14.511376 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:14.511380 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:27:14.511508 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 21:27:14.511562 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":370,"threads_started":1}
I20260927 21:27:14.512202 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:14.508567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40327 (local address 127.0.97.2:38511)
0927 21:27:14.508737 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:14.508746 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:14.509845 (+ 1099us) server_negotiation.cc:407] Connection header received
0927 21:27:14.510003 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:14.510011 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:14.510076 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:14.510181 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:14.511194 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:14.511199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:14.511205 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:27:14.511227 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:14.511262 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:27:14.511279 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:14.511397 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:14.511860 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 21:27:14.511926 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 21:27:14.512354 5205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef60bc6e2f041d58d919b87fdc44634" candidate_uuid: "7a96b7178c3a410c937d021de071a860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a68bc17903cb4f7ca236d4e9bc103a0f" is_pre_election: true
I20260927 21:27:14.512935 5205 raft_consensus.cc:2454] T 8ef60bc6e2f041d58d919b87fdc44634 P a68bc17903cb4f7ca236d4e9bc103a0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a96b7178c3a410c937d021de071a860 in term 0.
I20260927 21:27:14.513870 5401 leader_election.cc:302] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a96b7178c3a410c937d021de071a860, a68bc17903cb4f7ca236d4e9bc103a0f; no voters:
I20260927 21:27:14.514168 5336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef60bc6e2f041d58d919b87fdc44634" candidate_uuid: "7a96b7178c3a410c937d021de071a860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39c15b04b7164fb39bb8f2c795b1ef0e" is_pre_election: true
I20260927 21:27:14.514356 5537 raft_consensus.cc:2781] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:27:14.514503 5537 raft_consensus.cc:482] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:27:14.514613 5537 raft_consensus.cc:3037] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:14.514762 5336 raft_consensus.cc:2454] T 8ef60bc6e2f041d58d919b87fdc44634 P 39c15b04b7164fb39bb8f2c795b1ef0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a96b7178c3a410c937d021de071a860 in term 0.
I20260927 21:27:14.516973 5537 raft_consensus.cc:504] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a96b7178c3a410c937d021de071a860" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43469 } } peers { permanent_uuid: "a68bc17903cb4f7ca236d4e9bc103a0f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35675 } } peers { permanent_uuid: "39c15b04b7164fb39bb8f2c795b1ef0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38511 } }
I20260927 21:27:14.517685 5537 leader_election.cc:288] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860 [CANDIDATE]: Term 1 election: Requested vote from peers a68bc17903cb4f7ca236d4e9bc103a0f (127.0.97.1:35675), 39c15b04b7164fb39bb8f2c795b1ef0e (127.0.97.2:38511)
I20260927 21:27:14.518306 5205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef60bc6e2f041d58d919b87fdc44634" candidate_uuid: "7a96b7178c3a410c937d021de071a860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a68bc17903cb4f7ca236d4e9bc103a0f"
I20260927 21:27:14.518306 5336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef60bc6e2f041d58d919b87fdc44634" candidate_uuid: "7a96b7178c3a410c937d021de071a860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39c15b04b7164fb39bb8f2c795b1ef0e"
I20260927 21:27:14.518546 5336 raft_consensus.cc:3037] T 8ef60bc6e2f041d58d919b87fdc44634 P 39c15b04b7164fb39bb8f2c795b1ef0e [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:14.518543 5205 raft_consensus.cc:3037] T 8ef60bc6e2f041d58d919b87fdc44634 P a68bc17903cb4f7ca236d4e9bc103a0f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:14.521983 5336 raft_consensus.cc:2454] T 8ef60bc6e2f041d58d919b87fdc44634 P 39c15b04b7164fb39bb8f2c795b1ef0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a96b7178c3a410c937d021de071a860 in term 1.
I20260927 21:27:14.521983 5205 raft_consensus.cc:2454] T 8ef60bc6e2f041d58d919b87fdc44634 P a68bc17903cb4f7ca236d4e9bc103a0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a96b7178c3a410c937d021de071a860 in term 1.
I20260927 21:27:14.522665 5401 leader_election.cc:302] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 39c15b04b7164fb39bb8f2c795b1ef0e, 7a96b7178c3a410c937d021de071a860, a68bc17903cb4f7ca236d4e9bc103a0f; no voters:
I20260927 21:27:14.523092 5537 raft_consensus.cc:2781] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:27:14.523574 5537 raft_consensus.cc:686] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860 [term 1 LEADER]: Becoming Leader. State: Replica: 7a96b7178c3a410c937d021de071a860, State: Running, Role: LEADER
I20260927 21:27:14.524205 5537 consensus_queue.cc:237] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a96b7178c3a410c937d021de071a860" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43469 } } peers { permanent_uuid: "a68bc17903cb4f7ca236d4e9bc103a0f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35675 } } peers { permanent_uuid: "39c15b04b7164fb39bb8f2c795b1ef0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38511 } }
I20260927 21:27:14.528891 5062 catalog_manager.cc:5495] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a96b7178c3a410c937d021de071a860 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "7a96b7178c3a410c937d021de071a860" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a96b7178c3a410c937d021de071a860" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a68bc17903cb4f7ca236d4e9bc103a0f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39c15b04b7164fb39bb8f2c795b1ef0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38511 } health_report { overall_health: UNKNOWN } } }
W20260927 21:27:14.530366 5512 tablet.cc:2369] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:27:14.568356 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:14.566263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52340 (local address 127.0.97.3:43469)
0927 21:27:14.566458 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:14.566463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:14.566482 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:27:14.566641 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:14.566645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:14.566726 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:14.566824 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:14.567853 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:14.567857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:14.567861 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:27:14.567876 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:14.567906 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:27:14.567918 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:14.568038 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:14.568107 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 21:27:14.568153 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 21:27:14.574419 5382 tablet.cc:2369] T 8ef60bc6e2f041d58d919b87fdc44634 P 39c15b04b7164fb39bb8f2c795b1ef0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:27:14.577086 5205 raft_consensus.cc:1261] T 8ef60bc6e2f041d58d919b87fdc44634 P a68bc17903cb4f7ca236d4e9bc103a0f [term 1 FOLLOWER]: Refusing update from remote peer 7a96b7178c3a410c937d021de071a860: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:27:14.577386 5336 raft_consensus.cc:1261] T 8ef60bc6e2f041d58d919b87fdc44634 P 39c15b04b7164fb39bb8f2c795b1ef0e [term 1 FOLLOWER]: Refusing update from remote peer 7a96b7178c3a410c937d021de071a860: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:27:14.578476 5539 consensus_queue.cc:1035] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39c15b04b7164fb39bb8f2c795b1ef0e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:27:14.578861 5537 consensus_queue.cc:1035] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a68bc17903cb4f7ca236d4e9bc103a0f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:27:14.589643 5542 mvcc.cc:205] Tried to move back new op lower bound from 7334070004016926720 to 7334070003815424000. Current Snapshot: MvccSnapshot[applied={T|T < 7334070004016926720}]
I20260927 21:27:14.592865 5543 mvcc.cc:205] Tried to move back new op lower bound from 7334070004016926720 to 7334070003815424000. Current Snapshot: MvccSnapshot[applied={T|T < 7334070004016926720}]
I20260927 21:27:14.607916 5063 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:27:14.608253 5063 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:27:14.612178 5063 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06cd7cfb85ed4b30b915b8db13e202ab: Sending DeleteTablet for 3 replicas of tablet 8ef60bc6e2f041d58d919b87fdc44634
I20260927 21:27:14.613695 5316 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ef60bc6e2f041d58d919b87fdc44634 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:27:14 UTC) from {username='slave'} at 127.0.0.1:53796
I20260927 21:27:14.613673 5446 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ef60bc6e2f041d58d919b87fdc44634 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:27:14 UTC) from {username='slave'} at 127.0.0.1:52326
I20260927 21:27:14.614506 5185 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ef60bc6e2f041d58d919b87fdc44634 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:27:14 UTC) from {username='slave'} at 127.0.0.1:49820
I20260927 21:27:14.614952 5551 tablet_replica.cc:330] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860: stopping tablet replica
I20260927 21:27:14.615458 5551 raft_consensus.cc:2229] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:27:14.615975 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 5123
I20260927 21:27:14.616417 5551 raft_consensus.cc:2258] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:27:14.617262 5552 tablet_replica.cc:330] T 8ef60bc6e2f041d58d919b87fdc44634 P 39c15b04b7164fb39bb8f2c795b1ef0e: stopping tablet replica
I20260927 21:27:14.617858 5552 raft_consensus.cc:2229] T 8ef60bc6e2f041d58d919b87fdc44634 P 39c15b04b7164fb39bb8f2c795b1ef0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:27:14.618423 5552 raft_consensus.cc:2258] T 8ef60bc6e2f041d58d919b87fdc44634 P 39c15b04b7164fb39bb8f2c795b1ef0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:27:14.619357 5553 tablet_replica.cc:330] T 8ef60bc6e2f041d58d919b87fdc44634 P a68bc17903cb4f7ca236d4e9bc103a0f: stopping tablet replica
I20260927 21:27:14.619843 5553 raft_consensus.cc:2229] T 8ef60bc6e2f041d58d919b87fdc44634 P a68bc17903cb4f7ca236d4e9bc103a0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:27:14.620369 5553 raft_consensus.cc:2258] T 8ef60bc6e2f041d58d919b87fdc44634 P a68bc17903cb4f7ca236d4e9bc103a0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:27:14.620965 5552 ts_tablet_manager.cc:1905] T 8ef60bc6e2f041d58d919b87fdc44634 P 39c15b04b7164fb39bb8f2c795b1ef0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:27:14.621587 5551 ts_tablet_manager.cc:1905] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:27:14.628582 5552 ts_tablet_manager.cc:1918] T 8ef60bc6e2f041d58d919b87fdc44634 P 39c15b04b7164fb39bb8f2c795b1ef0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:27:14.628787 5552 log.cc:1196] T 8ef60bc6e2f041d58d919b87fdc44634 P 39c15b04b7164fb39bb8f2c795b1ef0e: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-1/wal/wals/8ef60bc6e2f041d58d919b87fdc44634
I20260927 21:27:14.629314 5551 ts_tablet_manager.cc:1918] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:27:14.629361 5552 ts_tablet_manager.cc:1939] T 8ef60bc6e2f041d58d919b87fdc44634 P 39c15b04b7164fb39bb8f2c795b1ef0e: Deleting consensus metadata
I20260927 21:27:14.629513 5551 log.cc:1196] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790544390748963-388-0/minicluster-data/ts-2/wal/wals/8ef60bc6e2f041d58d919b87fdc44634
I20260927 21:27:14.630105 5551 ts_tablet_manager.cc:1939] T 8ef60bc6e2f041d58d919b87fdc44634 P 7a96b7178c3a410c937d021de071a860: Deleting consensus metadata
I20260927 21:27:14.630522 5553 ts_tablet_manager.cc:1905] T 8ef60bc6e2f041d58d919b87fdc44634 P a68bc17903cb4f7ca236d4e9bc103a0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:27:14.631501 5048 catalog_manager.cc:4841] TS 39c15b04b7164fb39bb8f2c795b1ef0e (127.0.97.2:38511): tablet 8ef60bc6e2f041d58d919b87fdc44634 (table test-table [id=7c4c816ecd574a71a99ba7875947eab0]) successfully deleted
I20260927 21:27:14.631788 5051 catalog_manager.cc:4841] TS 7a96b7178c3a410c937d021de071a860 (127.0.97.3:43469): tablet 8ef60bc6e2f041d58d919b87fdc44634 (table test-table [id=7c4c816ecd574a71a99ba7875947eab0]) successfully deleted
I20260927 21:27:14.635042 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:14.618492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49828 (local address 127.0.97.1:35675)
0927 21:27:14.618729 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:14.618738 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:14.632929 (+ 14191us) server_negotiation.cc:407] Connection header received
0927 21:27:14.633081 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:14.633084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:14.633153 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:14.633287 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:14.634305 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:14.634310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:14.634317 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:27:14.634338 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:14.634384 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:27:14.634397 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:14.634563 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:14.634628 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 21:27:14.634698 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 21:27:14.635722 5245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:14.725515 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 5123
W20260927 21:27:14.749020 5050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:35675 (error 108)
I20260927 21:27:14.749420 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 5254
I20260927 21:27:14.751273 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:14.750779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35675 (local address 127.0.0.1:49836)
0927 21:27:14.750969 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:14.751107 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 21:27:14.751688 5050 catalog_manager.cc:4584] TS a68bc17903cb4f7ca236d4e9bc103a0f (127.0.97.1:35675): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ef60bc6e2f041d58d919b87fdc44634: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35675: connect: Connection refused (error 111)
I20260927 21:27:14.753085 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:14.750334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53810 (local address 127.0.97.2:38511)
0927 21:27:14.750545 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:14.750552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:14.750747 (+ 195us) server_negotiation.cc:407] Connection header received
0927 21:27:14.751061 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:14.751066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:14.751169 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:14.751244 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:14.752408 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:14.752414 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:14.752418 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:27:14.752438 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:14.752476 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:27:14.752488 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:14.752629 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:14.752762 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 21:27:14.752821 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 21:27:14.753877 5376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:14.805508 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:14.805016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35675 (local address 127.0.0.1:49850)
0927 21:27:14.805196 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:14.805309 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 21:27:14.806891 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:14.806423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35675 (local address 127.0.0.1:49852)
0927 21:27:14.806638 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:14.806736 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 21:27:14.843765 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 5254
I20260927 21:27:14.861398 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 5384
I20260927 21:27:14.864604 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:14.862333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52352 (local address 127.0.97.3:43469)
0927 21:27:14.862515 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:14.862523 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:14.862547 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:27:14.862799 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:14.862806 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:14.862862 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:14.862992 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:14.863913 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:14.863919 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:14.863923 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:27:14.863944 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:14.863982 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:27:14.863994 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:14.864173 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:14.864222 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 21:27:14.864282 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 21:27:14.865332 5506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:14.868203 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:14.867533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35675 (local address 127.0.0.1:49854)
0927 21:27:14.867829 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:14.867955 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260927 21:27:14.869681 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:14.869251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35675 (local address 127.0.0.1:49870)
0927 21:27:14.869444 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:14.869516 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 21:27:14.950311 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:14.949851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35675 (local address 127.0.0.1:49878)
0927 21:27:14.950056 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:14.950159 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 21:27:14.951486 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:14.951050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35675 (local address 127.0.0.1:49894)
0927 21:27:14.951277 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:14.951358 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 21:27:14.985961 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 5384
I20260927 21:27:15.004680 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 5033
I20260927 21:27:15.005867 5094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:15.113548 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 5033
2026-09-27T21:27:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2823 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 21:27:15.156359 388 mini_oidc.cc:152] Starting JWKS server
I20260927 21:27:15.159142 388 webserver.cc:460] Webserver started at https://127.0.0.1:45461/ using document root <none> and password file <none>
I20260927 21:27:15.159478 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 21:27:15.160136 388 webserver.cc:460] Webserver started at http://127.0.0.1:35195/ using document root <none> and password file <none>
2026-09-27T21:27:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:27:15Z Disabled control of system clock
I20260927 21:27:15.189788 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35959
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35959
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45461/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544390748963-388-0/testchainca.cert with env {}
W20260927 21:27:15.527814 5565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:15.528132 5565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:15.528219 5565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:15.535698 5565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:27:15.535825 5565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:15.535863 5565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:15.535912 5565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:27:15.535957 5565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 21:27:15.536018 5565 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 21:27:15.536058 5565 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45461/jwks.json
I20260927 21:27:15.546298 5565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35959
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45461/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544390748963-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:15.547669 5565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:15.548969 5565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:15.561188 5570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:15.561272 5573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:15.561299 5571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:15.562500 5565 server_base.cc:1034] running on GCE node
I20260927 21:27:15.563458 5565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:15.564965 5565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:15.566249 5565 hybrid_clock.cc:648] HybridClock initialized: now 1790544435566187 us; error 51 us; skew 500 ppm
I20260927 21:27:15.566715 5565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:27:15.569381 5565 webserver.cc:460] Webserver started at http://127.0.97.62:38763/ using document root <none> and password file <none>
I20260927 21:27:15.570235 5565 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:15.570387 5565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:15.570724 5565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:15.574152 5565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "e5f3bb61d81f4e46a18f0c7aca646133"
format_stamp: "Formatted at 2026-09-27 21:27:15 on dist-test-slave-bxm0"
server_key: "928f092e424d655f4f3b513faca7a162"
server_key_iv: "50b367346cc450974f1699ffddbf8f20"
server_key_version: "encryptionkey@0"
I20260927 21:27:15.575059 5565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "e5f3bb61d81f4e46a18f0c7aca646133"
format_stamp: "Formatted at 2026-09-27 21:27:15 on dist-test-slave-bxm0"
server_key: "928f092e424d655f4f3b513faca7a162"
server_key_iv: "50b367346cc450974f1699ffddbf8f20"
server_key_version: "encryptionkey@0"
I20260927 21:27:15.581147 5565 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 21:27:15.585269 5579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:15.586998 5565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 21:27:15.587221 5565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "e5f3bb61d81f4e46a18f0c7aca646133"
format_stamp: "Formatted at 2026-09-27 21:27:15 on dist-test-slave-bxm0"
server_key: "928f092e424d655f4f3b513faca7a162"
server_key_iv: "50b367346cc450974f1699ffddbf8f20"
server_key_version: "encryptionkey@0"
I20260927 21:27:15.587483 5565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:15.609917 5565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:15.610843 5565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:15.611157 5565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:15.635479 5565 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35959
I20260927 21:27:15.635522 5630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35959 every 8 connection(s)
I20260927 21:27:15.636957 5565 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:27:15.641088 5631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:15.641423 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 5565
I20260927 21:27:15.641651 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:27:15.642333 388 external_mini_cluster.cc:1428] Setting key b8a5230468674f7565117b15868d8b48
I20260927 21:27:15.656615 5631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f3bb61d81f4e46a18f0c7aca646133: Bootstrap starting.
I20260927 21:27:15.659462 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:15.644617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42588 (local address 127.0.97.62:35959)
0927 21:27:15.645879 (+ 1262us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:15.645905 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:15.645965 (+ 60us) server_negotiation.cc:407] Connection header received
0927 21:27:15.648273 (+ 2308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:15.648307 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:15.649052 (+ 745us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:15.649570 (+ 518us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:15.650197 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:15.652137 (+ 1940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:15.653656 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:15.654802 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:15.655004 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:15.656317 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:15.656328 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:15.656349 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:27:15.657128 (+ 779us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:15.657239 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:27:15.657265 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:15.657379 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:15.657681 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 21:27:15.658033 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":412,"threads_started":1}
I20260927 21:27:15.660723 5631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5f3bb61d81f4e46a18f0c7aca646133: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:15.662151 5631 log.cc:824] T 00000000000000000000000000000000 P e5f3bb61d81f4e46a18f0c7aca646133: Log is configured to *not* fsync() on all Append() calls
I20260927 21:27:15.664070 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:27:15.665733 5631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f3bb61d81f4e46a18f0c7aca646133: No bootstrap required, opened a new log
I20260927 21:27:15.673718 5631 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5f3bb61d81f4e46a18f0c7aca646133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f3bb61d81f4e46a18f0c7aca646133" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35959 } }
I20260927 21:27:15.674177 5631 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5f3bb61d81f4e46a18f0c7aca646133 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:15.674299 5631 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5f3bb61d81f4e46a18f0c7aca646133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5f3bb61d81f4e46a18f0c7aca646133, State: Initialized, Role: FOLLOWER
I20260927 21:27:15.675077 5631 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5f3bb61d81f4e46a18f0c7aca646133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f3bb61d81f4e46a18f0c7aca646133" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35959 } }
I20260927 21:27:15.675302 5631 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5f3bb61d81f4e46a18f0c7aca646133 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:27:15.675405 5631 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5f3bb61d81f4e46a18f0c7aca646133 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:27:15.675524 5631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5f3bb61d81f4e46a18f0c7aca646133 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:15.677731 5631 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5f3bb61d81f4e46a18f0c7aca646133 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f3bb61d81f4e46a18f0c7aca646133" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35959 } }
I20260927 21:27:15.678323 5631 leader_election.cc:302] T 00000000000000000000000000000000 P e5f3bb61d81f4e46a18f0c7aca646133 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5f3bb61d81f4e46a18f0c7aca646133; no voters:
I20260927 21:27:15.678905 5631 leader_election.cc:288] T 00000000000000000000000000000000 P e5f3bb61d81f4e46a18f0c7aca646133 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:27:15.679114 5636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5f3bb61d81f4e46a18f0c7aca646133 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:27:15.680346 5636 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5f3bb61d81f4e46a18f0c7aca646133 [term 1 LEADER]: Becoming Leader. State: Replica: e5f3bb61d81f4e46a18f0c7aca646133, State: Running, Role: LEADER
I20260927 21:27:15.680938 5636 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5f3bb61d81f4e46a18f0c7aca646133 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f3bb61d81f4e46a18f0c7aca646133" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35959 } }
I20260927 21:27:15.681295 5631 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5f3bb61d81f4e46a18f0c7aca646133 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:27:15.684121 5637 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f3bb61d81f4e46a18f0c7aca646133 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5f3bb61d81f4e46a18f0c7aca646133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f3bb61d81f4e46a18f0c7aca646133" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35959 } } }
I20260927 21:27:15.684129 5638 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f3bb61d81f4e46a18f0c7aca646133 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5f3bb61d81f4e46a18f0c7aca646133. Latest consensus state: current_term: 1 leader_uuid: "e5f3bb61d81f4e46a18f0c7aca646133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f3bb61d81f4e46a18f0c7aca646133" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35959 } } }
I20260927 21:27:15.684652 5637 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f3bb61d81f4e46a18f0c7aca646133 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:15.684690 5638 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f3bb61d81f4e46a18f0c7aca646133 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:15.688977 5645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:27:15.693876 5645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:27:15.703320 5645 catalog_manager.cc:1347] Generated new cluster ID: 22896cbc783e42f2a9f786b6bfb763bd
I20260927 21:27:15.703456 5645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:27:15.726800 5645 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:27:15.728199 5645 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:27:15.735854 5645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5f3bb61d81f4e46a18f0c7aca646133: Generated new TSK 0
I20260927 21:27:15.736763 5645 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 21:27:15.784811 5661 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 21:27:15.785079 5661 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 21:27:15.786365 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:15.763769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42590 (local address 127.0.97.62:35959)
0927 21:27:15.763977 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:15.763984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:15.764406 (+ 422us) server_negotiation.cc:407] Connection header received
0927 21:27:15.764716 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:15.764721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:15.764818 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:15.764993 (+ 175us) server_negotiation.cc:226] Negotiated authn=JWT
0927 21:27:15.765515 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:15.766688 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:15.768083 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:15.768775 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:15.768919 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:15.769450 (+ 531us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 21:27:15.785544 (+ 16094us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 21:27:15.786009 (+ 465us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42590: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 21:27:15.786497 5634 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42590: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 21:27:15.787716 5660 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:15.763567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35959 (local address 127.0.0.1:42590)
0927 21:27:15.764288 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:15.764305 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:15.764464 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:15.765025 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:15.765028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:15.765038 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0927 21:27:15.765369 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:15.765377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:15.766835 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:15.766838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:15.767939 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:15.767951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:15.768893 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:15.768948 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:15.769097 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:15.769161 (+ 64us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 21:27:15.785903 (+ 16742us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 21:27:15.786196 (+ 293us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:35959: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":396,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:35959: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 21:27:15.790277 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 5565
I20260927 21:27:15.791795 5626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:15.886724 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 5565
2026-09-27T21:27:15Z chronyd exiting
I20260927 21:27:15.938066 388 test_util.cc:175] -----------------------------------------------
I20260927 21:27:15.938254 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790544390748963-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (786 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 21:27:15.942152 388 mini_oidc.cc:152] Starting JWKS server
I20260927 21:27:15.944002 388 webserver.cc:460] Webserver started at https://127.0.0.1:37771/ using document root <none> and password file <none>
I20260927 21:27:15.944281 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 21:27:15.944957 388 webserver.cc:460] Webserver started at http://127.0.0.1:36213/ using document root <none> and password file <none>
2026-09-27T21:27:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:27:15Z Disabled control of system clock
I20260927 21:27:15.972095 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39055
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39055
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37771/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544390748963-388-0/testchainca.cert with env {}
W20260927 21:27:16.308678 5668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:16.309074 5668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:16.309247 5668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:16.317210 5668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:27:16.317395 5668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:16.317467 5668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:16.317555 5668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:27:16.317670 5668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 21:27:16.317762 5668 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 21:27:16.317822 5668 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37771/jwks.json
I20260927 21:27:16.328565 5668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39055
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37771/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544390748963-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:16.330045 5668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:16.331406 5668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:16.343658 5673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:16.343604 5674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:16.343618 5676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:16.344527 5668 server_base.cc:1034] running on GCE node
I20260927 21:27:16.345428 5668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:16.347096 5668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:16.348335 5668 hybrid_clock.cc:648] HybridClock initialized: now 1790544436348283 us; error 60 us; skew 500 ppm
I20260927 21:27:16.348694 5668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:27:16.355015 5668 webserver.cc:460] Webserver started at http://127.0.97.62:34145/ using document root <none> and password file <none>
I20260927 21:27:16.355816 5668 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:16.355969 5668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:16.356314 5668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:16.359377 5668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "53ab6f6e7ab34f679e8cc7ffa444935b"
format_stamp: "Formatted at 2026-09-27 21:27:16 on dist-test-slave-bxm0"
server_key: "5f3e474803bca63a92e325381f2d3f3a"
server_key_iv: "de11ba8de2985281a49bbc9bcc98d36a"
server_key_version: "encryptionkey@0"
I20260927 21:27:16.360250 5668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "53ab6f6e7ab34f679e8cc7ffa444935b"
format_stamp: "Formatted at 2026-09-27 21:27:16 on dist-test-slave-bxm0"
server_key: "5f3e474803bca63a92e325381f2d3f3a"
server_key_iv: "de11ba8de2985281a49bbc9bcc98d36a"
server_key_version: "encryptionkey@0"
I20260927 21:27:16.365921 5668 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 21:27:16.369668 5682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:16.371212 5668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:27:16.371423 5668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "53ab6f6e7ab34f679e8cc7ffa444935b"
format_stamp: "Formatted at 2026-09-27 21:27:16 on dist-test-slave-bxm0"
server_key: "5f3e474803bca63a92e325381f2d3f3a"
server_key_iv: "de11ba8de2985281a49bbc9bcc98d36a"
server_key_version: "encryptionkey@0"
I20260927 21:27:16.371655 5668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:16.388754 5668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:16.389741 5668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:16.390033 5668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:16.409235 5668 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39055
I20260927 21:27:16.409263 5733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39055 every 8 connection(s)
I20260927 21:27:16.410674 5668 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:27:16.413484 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 5668
I20260927 21:27:16.413703 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:27:16.414507 388 external_mini_cluster.cc:1428] Setting key 75146d6229968c10b8c90f1235071510
I20260927 21:27:16.415256 5734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:16.428519 5734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53ab6f6e7ab34f679e8cc7ffa444935b: Bootstrap starting.
I20260927 21:27:16.429137 5737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:16.416566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38612 (local address 127.0.97.62:39055)
0927 21:27:16.417567 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:16.417579 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:16.417692 (+ 113us) server_negotiation.cc:407] Connection header received
0927 21:27:16.420096 (+ 2404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:16.420125 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:16.420701 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:16.421180 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:16.421850 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:16.423208 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:16.424523 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:16.425270 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:16.425496 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:16.426759 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:16.426767 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:16.426783 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:27:16.427332 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:16.427413 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:27:16.427433 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:16.427578 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:16.427792 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 21:27:16.428117 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":356,"threads_started":1}
I20260927 21:27:16.432251 5734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53ab6f6e7ab34f679e8cc7ffa444935b: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:16.433115 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:27:16.433459 5734 log.cc:824] T 00000000000000000000000000000000 P 53ab6f6e7ab34f679e8cc7ffa444935b: Log is configured to *not* fsync() on all Append() calls
I20260927 21:27:16.436604 5734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53ab6f6e7ab34f679e8cc7ffa444935b: No bootstrap required, opened a new log
I20260927 21:27:16.444109 5734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53ab6f6e7ab34f679e8cc7ffa444935b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ab6f6e7ab34f679e8cc7ffa444935b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39055 } }
I20260927 21:27:16.444550 5734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53ab6f6e7ab34f679e8cc7ffa444935b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:16.444662 5734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53ab6f6e7ab34f679e8cc7ffa444935b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53ab6f6e7ab34f679e8cc7ffa444935b, State: Initialized, Role: FOLLOWER
I20260927 21:27:16.445308 5734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53ab6f6e7ab34f679e8cc7ffa444935b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ab6f6e7ab34f679e8cc7ffa444935b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39055 } }
I20260927 21:27:16.445550 5734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53ab6f6e7ab34f679e8cc7ffa444935b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:27:16.445719 5734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53ab6f6e7ab34f679e8cc7ffa444935b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:27:16.445904 5734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53ab6f6e7ab34f679e8cc7ffa444935b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:16.448414 5734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53ab6f6e7ab34f679e8cc7ffa444935b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ab6f6e7ab34f679e8cc7ffa444935b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39055 } }
I20260927 21:27:16.448976 5734 leader_election.cc:302] T 00000000000000000000000000000000 P 53ab6f6e7ab34f679e8cc7ffa444935b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53ab6f6e7ab34f679e8cc7ffa444935b; no voters:
I20260927 21:27:16.449559 5734 leader_election.cc:288] T 00000000000000000000000000000000 P 53ab6f6e7ab34f679e8cc7ffa444935b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:27:16.449774 5739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53ab6f6e7ab34f679e8cc7ffa444935b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:27:16.450568 5739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53ab6f6e7ab34f679e8cc7ffa444935b [term 1 LEADER]: Becoming Leader. State: Replica: 53ab6f6e7ab34f679e8cc7ffa444935b, State: Running, Role: LEADER
I20260927 21:27:16.451148 5739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53ab6f6e7ab34f679e8cc7ffa444935b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ab6f6e7ab34f679e8cc7ffa444935b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39055 } }
I20260927 21:27:16.451937 5734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53ab6f6e7ab34f679e8cc7ffa444935b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:27:16.454120 5740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53ab6f6e7ab34f679e8cc7ffa444935b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53ab6f6e7ab34f679e8cc7ffa444935b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ab6f6e7ab34f679e8cc7ffa444935b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39055 } } }
I20260927 21:27:16.454227 5741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53ab6f6e7ab34f679e8cc7ffa444935b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53ab6f6e7ab34f679e8cc7ffa444935b. Latest consensus state: current_term: 1 leader_uuid: "53ab6f6e7ab34f679e8cc7ffa444935b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ab6f6e7ab34f679e8cc7ffa444935b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39055 } } }
I20260927 21:27:16.454524 5740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53ab6f6e7ab34f679e8cc7ffa444935b [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:16.454633 5741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53ab6f6e7ab34f679e8cc7ffa444935b [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:16.457166 5744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:27:16.463661 5744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:27:16.473539 5744 catalog_manager.cc:1347] Generated new cluster ID: f56e75f50f034dbe862b6a40fe551d96
I20260927 21:27:16.473740 5744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:27:16.485111 5744 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:27:16.487200 5744 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:27:16.498898 5744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53ab6f6e7ab34f679e8cc7ffa444935b: Generated new TSK 0
I20260927 21:27:16.500111 5744 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 21:27:16.544857 5764 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 21:27:16.544971 5764 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 21:27:16.546386 5737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:16.524959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38626 (local address 127.0.97.62:39055)
0927 21:27:16.525160 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:16.525173 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:16.525746 (+ 573us) server_negotiation.cc:407] Connection header received
0927 21:27:16.526049 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:16.526053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:16.526124 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:16.526262 (+ 138us) server_negotiation.cc:226] Negotiated authn=JWT
0927 21:27:16.526921 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:16.527712 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:16.529820 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:16.530297 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:16.530406 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:16.531042 (+ 636us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 21:27:16.545576 (+ 14534us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37771/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 21:27:16.546051 (+ 475us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38626: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37771/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 21:27:16.546495 5763 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:16.524731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39055 (local address 127.0.0.1:38626)
0927 21:27:16.525594 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:16.525643 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:16.525825 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:16.526310 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:16.526316 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:16.526330 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0927 21:27:16.526664 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:16.526678 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:16.527874 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:16.527880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:16.529662 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:16.529679 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:16.530436 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:16.530489 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:16.530694 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:16.530744 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 21:27:16.545937 (+ 15193us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37771/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 21:27:16.546160 (+ 223us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:39055: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37771/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":552,"threads_started":1}
W20260927 21:27:16.546523 5737 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38626: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37771/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260927 21:27:16.547538 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 5668
I20260927 21:27:16.548688 5729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:16.642877 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 5668
2026-09-27T21:27:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (758 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 21:27:16.701116 388 mini_oidc.cc:152] Starting JWKS server
I20260927 21:27:16.703208 388 webserver.cc:460] Webserver started at https://127.0.0.1:44343/ using document root <none> and password file <none>
I20260927 21:27:16.703492 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 21:27:16.704201 388 webserver.cc:460] Webserver started at http://127.0.0.1:33483/ using document root <none> and password file <none>
2026-09-27T21:27:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:27:16Z Disabled control of system clock
I20260927 21:27:16.733026 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34675
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34675
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44343/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 21:27:17.068617 5771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:17.068955 5771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:17.069051 5771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:17.076858 5771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:27:17.076996 5771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:17.077034 5771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:17.077088 5771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:27:17.077133 5771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 21:27:17.077188 5771 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 21:27:17.077227 5771 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44343/jwks.json
I20260927 21:27:17.087059 5771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34675
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44343/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:17.088325 5771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:17.090173 5771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:17.103024 5779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:17.103081 5777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:17.102993 5776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:17.104190 5771 server_base.cc:1034] running on GCE node
I20260927 21:27:17.105073 5771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:17.106873 5771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:17.108085 5771 hybrid_clock.cc:648] HybridClock initialized: now 1790544437108043 us; error 35 us; skew 500 ppm
I20260927 21:27:17.108446 5771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:27:17.110909 5771 webserver.cc:460] Webserver started at http://127.0.97.62:40027/ using document root <none> and password file <none>
I20260927 21:27:17.111635 5771 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:17.111724 5771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:17.112098 5771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:17.115109 5771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "c29fc3a9a3704030b9b9cce461cd5101"
format_stamp: "Formatted at 2026-09-27 21:27:17 on dist-test-slave-bxm0"
server_key: "335c9325bb1819703bd29b8b646d5c8a"
server_key_iv: "28cbef6bf08532d0d2d45cd5f4b6ebc2"
server_key_version: "encryptionkey@0"
I20260927 21:27:17.115922 5771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "c29fc3a9a3704030b9b9cce461cd5101"
format_stamp: "Formatted at 2026-09-27 21:27:17 on dist-test-slave-bxm0"
server_key: "335c9325bb1819703bd29b8b646d5c8a"
server_key_iv: "28cbef6bf08532d0d2d45cd5f4b6ebc2"
server_key_version: "encryptionkey@0"
I20260927 21:27:17.121266 5771 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 21:27:17.125140 5785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:17.126715 5771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 21:27:17.126928 5771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "c29fc3a9a3704030b9b9cce461cd5101"
format_stamp: "Formatted at 2026-09-27 21:27:17 on dist-test-slave-bxm0"
server_key: "335c9325bb1819703bd29b8b646d5c8a"
server_key_iv: "28cbef6bf08532d0d2d45cd5f4b6ebc2"
server_key_version: "encryptionkey@0"
I20260927 21:27:17.127173 5771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:17.142856 5771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:17.143755 5771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:17.144073 5771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:17.163195 5771 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34675
I20260927 21:27:17.163214 5836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34675 every 8 connection(s)
I20260927 21:27:17.164611 5771 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:27:17.168651 5837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:17.174180 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 5771
I20260927 21:27:17.174401 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:27:17.175117 388 external_mini_cluster.cc:1428] Setting key 1976b90f9132335a11f8b1a14e4776a0
I20260927 21:27:17.180680 5837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c29fc3a9a3704030b9b9cce461cd5101: Bootstrap starting.
I20260927 21:27:17.184572 5837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c29fc3a9a3704030b9b9cce461cd5101: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:17.185971 5837 log.cc:824] T 00000000000000000000000000000000 P c29fc3a9a3704030b9b9cce461cd5101: Log is configured to *not* fsync() on all Append() calls
I20260927 21:27:17.189497 5837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c29fc3a9a3704030b9b9cce461cd5101: No bootstrap required, opened a new log
I20260927 21:27:17.190034 5840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:17.177221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44432 (local address 127.0.97.62:34675)
0927 21:27:17.178417 (+ 1196us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:17.178433 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:17.178481 (+ 48us) server_negotiation.cc:407] Connection header received
0927 21:27:17.180192 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:17.180216 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:17.180694 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:17.181105 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:17.181737 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:17.183089 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:17.184996 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:17.185882 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:17.186053 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:17.187371 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:17.187385 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:17.187409 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:27:17.188011 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:17.188090 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:27:17.188112 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:17.188212 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:17.188473 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 21:27:17.188853 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":448,"threads_started":1}
I20260927 21:27:17.194379 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:27:17.198004 5837 raft_consensus.cc:348] T 00000000000000000000000000000000 P c29fc3a9a3704030b9b9cce461cd5101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c29fc3a9a3704030b9b9cce461cd5101" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34675 } }
I20260927 21:27:17.198443 5837 raft_consensus.cc:374] T 00000000000000000000000000000000 P c29fc3a9a3704030b9b9cce461cd5101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:17.198544 5837 raft_consensus.cc:729] T 00000000000000000000000000000000 P c29fc3a9a3704030b9b9cce461cd5101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c29fc3a9a3704030b9b9cce461cd5101, State: Initialized, Role: FOLLOWER
I20260927 21:27:17.199234 5837 consensus_queue.cc:260] T 00000000000000000000000000000000 P c29fc3a9a3704030b9b9cce461cd5101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c29fc3a9a3704030b9b9cce461cd5101" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34675 } }
I20260927 21:27:17.199458 5837 raft_consensus.cc:388] T 00000000000000000000000000000000 P c29fc3a9a3704030b9b9cce461cd5101 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:27:17.199642 5837 raft_consensus.cc:482] T 00000000000000000000000000000000 P c29fc3a9a3704030b9b9cce461cd5101 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:27:17.199834 5837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c29fc3a9a3704030b9b9cce461cd5101 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:17.202724 5837 raft_consensus.cc:504] T 00000000000000000000000000000000 P c29fc3a9a3704030b9b9cce461cd5101 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c29fc3a9a3704030b9b9cce461cd5101" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34675 } }
I20260927 21:27:17.203339 5837 leader_election.cc:302] T 00000000000000000000000000000000 P c29fc3a9a3704030b9b9cce461cd5101 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c29fc3a9a3704030b9b9cce461cd5101; no voters:
I20260927 21:27:17.204005 5837 leader_election.cc:288] T 00000000000000000000000000000000 P c29fc3a9a3704030b9b9cce461cd5101 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:27:17.204149 5842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c29fc3a9a3704030b9b9cce461cd5101 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:27:17.205585 5842 raft_consensus.cc:686] T 00000000000000000000000000000000 P c29fc3a9a3704030b9b9cce461cd5101 [term 1 LEADER]: Becoming Leader. State: Replica: c29fc3a9a3704030b9b9cce461cd5101, State: Running, Role: LEADER
I20260927 21:27:17.206161 5842 consensus_queue.cc:237] T 00000000000000000000000000000000 P c29fc3a9a3704030b9b9cce461cd5101 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c29fc3a9a3704030b9b9cce461cd5101" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34675 } }
I20260927 21:27:17.206522 5837 sys_catalog.cc:564] T 00000000000000000000000000000000 P c29fc3a9a3704030b9b9cce461cd5101 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:27:17.209059 5844 sys_catalog.cc:455] T 00000000000000000000000000000000 P c29fc3a9a3704030b9b9cce461cd5101 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c29fc3a9a3704030b9b9cce461cd5101. Latest consensus state: current_term: 1 leader_uuid: "c29fc3a9a3704030b9b9cce461cd5101" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c29fc3a9a3704030b9b9cce461cd5101" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34675 } } }
I20260927 21:27:17.209434 5844 sys_catalog.cc:458] T 00000000000000000000000000000000 P c29fc3a9a3704030b9b9cce461cd5101 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:17.209178 5843 sys_catalog.cc:455] T 00000000000000000000000000000000 P c29fc3a9a3704030b9b9cce461cd5101 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c29fc3a9a3704030b9b9cce461cd5101" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c29fc3a9a3704030b9b9cce461cd5101" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34675 } } }
I20260927 21:27:17.209684 5843 sys_catalog.cc:458] T 00000000000000000000000000000000 P c29fc3a9a3704030b9b9cce461cd5101 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:17.212857 5850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:27:17.218330 5850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:27:17.229133 5850 catalog_manager.cc:1347] Generated new cluster ID: 5d5b05ef90f44838b63fd4b295b39be1
I20260927 21:27:17.229305 5850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:27:17.246490 5850 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:27:17.248524 5850 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:27:17.259469 5850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c29fc3a9a3704030b9b9cce461cd5101: Generated new TSK 0
I20260927 21:27:17.260787 5850 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:27:17.284334 5840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:17.275107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44436 (local address 127.0.97.62:34675)
0927 21:27:17.275329 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:17.275337 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:17.275819 (+ 482us) server_negotiation.cc:407] Connection header received
0927 21:27:17.276047 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:17.276054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:17.276165 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:17.276361 (+ 196us) server_negotiation.cc:226] Negotiated authn=JWT
0927 21:27:17.276841 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:17.278011 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:17.279403 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:17.280073 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:17.280216 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:17.280829 (+ 613us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 21:27:17.283358 (+ 2529us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44343/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 21:27:17.283863 (+ 505us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44436: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44343/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 21:27:17.284499 5840 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44436: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44343/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 21:27:17.285835 5866 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:17.275044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34675 (local address 127.0.0.1:44436)
0927 21:27:17.275704 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:17.275721 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:17.275859 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:17.276399 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:17.276402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:17.276412 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0927 21:27:17.276686 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:17.276694 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:17.278166 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:17.278170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:17.279251 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:17.279262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:17.280254 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:17.280326 (+ 72us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:17.280492 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:17.280540 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 21:27:17.283680 (+ 3140us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44343/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 21:27:17.285364 (+ 1684us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:34675: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44343/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":1327,"thread_start_us":365,"threads_started":1}
I20260927 21:27:17.286966 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 5771
I20260927 21:27:17.288322 5832 generic_service.cc:196] Checking for leaks (request via RPC)
E20260927 21:27:17.299104 5867 webserver.cc:533] Webserver: sslize failed:
E20260927 21:27:17.299249 5867 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 21:27:17.384212 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 5771
2026-09-27T21:27:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (739 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T21:27:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:27:17Z Disabled control of system clock
I20260927 21:27:17.468465 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33481
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33481
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544390748963-388-0/testchainca.cert with env {}
W20260927 21:27:17.812289 5872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:17.812649 5872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:17.812803 5872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:17.820339 5872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:27:17.820482 5872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:17.820530 5872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:17.820654 5872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:27:17.820742 5872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 21:27:17.820856 5872 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 21:27:17.820915 5872 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 21:27:17.831434 5872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33481
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544390748963-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:17.832931 5872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:17.834344 5872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:17.847106 5878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:17.847074 5880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:17.847925 5877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:17.848367 5872 server_base.cc:1034] running on GCE node
I20260927 21:27:17.849311 5872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:17.850968 5872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:17.852257 5872 hybrid_clock.cc:648] HybridClock initialized: now 1790544437852211 us; error 45 us; skew 500 ppm
I20260927 21:27:17.852742 5872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:27:17.855397 5872 webserver.cc:460] Webserver started at http://127.0.97.62:36983/ using document root <none> and password file <none>
I20260927 21:27:17.856168 5872 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:17.856313 5872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:17.856663 5872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:17.859815 5872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "feae40d2279a43c388507223e609d804"
format_stamp: "Formatted at 2026-09-27 21:27:17 on dist-test-slave-bxm0"
server_key: "485906bc58c6e70186e889ec4d279a22"
server_key_iv: "9e74c5ed366c7062934d26b521db49ff"
server_key_version: "encryptionkey@0"
I20260927 21:27:17.860693 5872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "feae40d2279a43c388507223e609d804"
format_stamp: "Formatted at 2026-09-27 21:27:17 on dist-test-slave-bxm0"
server_key: "485906bc58c6e70186e889ec4d279a22"
server_key_iv: "9e74c5ed366c7062934d26b521db49ff"
server_key_version: "encryptionkey@0"
I20260927 21:27:17.866246 5872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 21:27:17.870162 5886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:17.871711 5872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:27:17.871949 5872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "feae40d2279a43c388507223e609d804"
format_stamp: "Formatted at 2026-09-27 21:27:17 on dist-test-slave-bxm0"
server_key: "485906bc58c6e70186e889ec4d279a22"
server_key_iv: "9e74c5ed366c7062934d26b521db49ff"
server_key_version: "encryptionkey@0"
I20260927 21:27:17.872193 5872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:17.885830 5872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:17.886771 5872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:17.887084 5872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:17.906422 5872 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33481
I20260927 21:27:17.906450 5937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33481 every 8 connection(s)
I20260927 21:27:17.907935 5872 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:27:17.910151 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 5872
I20260927 21:27:17.910363 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:27:17.911213 388 external_mini_cluster.cc:1428] Setting key 62732c9672eccd2bacc2a3c6670db008
I20260927 21:27:17.912904 5938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:17.925511 5941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:17.913343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41440 (local address 127.0.97.62:33481)
0927 21:27:17.914423 (+ 1080us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:17.914436 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:17.914480 (+ 44us) server_negotiation.cc:407] Connection header received
0927 21:27:17.916034 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:17.916058 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:17.916595 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:17.917053 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:17.917734 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:17.919177 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:17.920542 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:17.921285 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:17.921501 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:17.922894 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:17.922902 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:17.922932 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:27:17.923558 (+ 626us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:17.923654 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:27:17.923677 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:17.923826 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:17.924091 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 21:27:17.924450 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":394,"threads_started":1}
I20260927 21:27:17.925555 5938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feae40d2279a43c388507223e609d804: Bootstrap starting.
I20260927 21:27:17.929157 5938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P feae40d2279a43c388507223e609d804: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:17.929847 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:27:17.930424 5938 log.cc:824] T 00000000000000000000000000000000 P feae40d2279a43c388507223e609d804: Log is configured to *not* fsync() on all Append() calls
I20260927 21:27:17.933808 5938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feae40d2279a43c388507223e609d804: No bootstrap required, opened a new log
I20260927 21:27:17.941892 5938 raft_consensus.cc:348] T 00000000000000000000000000000000 P feae40d2279a43c388507223e609d804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feae40d2279a43c388507223e609d804" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33481 } }
I20260927 21:27:17.942447 5938 raft_consensus.cc:374] T 00000000000000000000000000000000 P feae40d2279a43c388507223e609d804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:17.942603 5938 raft_consensus.cc:729] T 00000000000000000000000000000000 P feae40d2279a43c388507223e609d804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feae40d2279a43c388507223e609d804, State: Initialized, Role: FOLLOWER
I20260927 21:27:17.943356 5938 consensus_queue.cc:260] T 00000000000000000000000000000000 P feae40d2279a43c388507223e609d804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feae40d2279a43c388507223e609d804" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33481 } }
I20260927 21:27:17.943580 5938 raft_consensus.cc:388] T 00000000000000000000000000000000 P feae40d2279a43c388507223e609d804 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:27:17.943683 5938 raft_consensus.cc:482] T 00000000000000000000000000000000 P feae40d2279a43c388507223e609d804 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:27:17.943812 5938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P feae40d2279a43c388507223e609d804 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:17.946081 5938 raft_consensus.cc:504] T 00000000000000000000000000000000 P feae40d2279a43c388507223e609d804 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feae40d2279a43c388507223e609d804" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33481 } }
I20260927 21:27:17.946619 5938 leader_election.cc:302] T 00000000000000000000000000000000 P feae40d2279a43c388507223e609d804 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: feae40d2279a43c388507223e609d804; no voters:
I20260927 21:27:17.947194 5938 leader_election.cc:288] T 00000000000000000000000000000000 P feae40d2279a43c388507223e609d804 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:27:17.947420 5943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P feae40d2279a43c388507223e609d804 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:27:17.948500 5943 raft_consensus.cc:686] T 00000000000000000000000000000000 P feae40d2279a43c388507223e609d804 [term 1 LEADER]: Becoming Leader. State: Replica: feae40d2279a43c388507223e609d804, State: Running, Role: LEADER
I20260927 21:27:17.949258 5943 consensus_queue.cc:237] T 00000000000000000000000000000000 P feae40d2279a43c388507223e609d804 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feae40d2279a43c388507223e609d804" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33481 } }
I20260927 21:27:17.949656 5938 sys_catalog.cc:564] T 00000000000000000000000000000000 P feae40d2279a43c388507223e609d804 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:27:17.952241 5945 sys_catalog.cc:455] T 00000000000000000000000000000000 P feae40d2279a43c388507223e609d804 [sys.catalog]: SysCatalogTable state changed. Reason: New leader feae40d2279a43c388507223e609d804. Latest consensus state: current_term: 1 leader_uuid: "feae40d2279a43c388507223e609d804" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feae40d2279a43c388507223e609d804" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33481 } } }
I20260927 21:27:17.952716 5945 sys_catalog.cc:458] T 00000000000000000000000000000000 P feae40d2279a43c388507223e609d804 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:17.952504 5944 sys_catalog.cc:455] T 00000000000000000000000000000000 P feae40d2279a43c388507223e609d804 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "feae40d2279a43c388507223e609d804" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feae40d2279a43c388507223e609d804" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33481 } } }
I20260927 21:27:17.952916 5944 sys_catalog.cc:458] T 00000000000000000000000000000000 P feae40d2279a43c388507223e609d804 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:17.954986 5952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:27:17.960613 5952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:27:17.970489 5952 catalog_manager.cc:1347] Generated new cluster ID: 720c867a6c2d4d4a8974c1645ff5a2e3
I20260927 21:27:17.970619 5952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:27:17.984674 5952 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:27:17.986169 5952 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:27:17.995365 5952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P feae40d2279a43c388507223e609d804: Generated new TSK 0
I20260927 21:27:17.996304 5952 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:27:18.098246 5941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:18.020807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41450 (local address 127.0.97.62:33481)
0927 21:27:18.021056 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:18.021068 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:18.021704 (+ 636us) server_negotiation.cc:407] Connection header received
0927 21:27:18.022005 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:18.022009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:18.022108 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:18.022273 (+ 165us) server_negotiation.cc:226] Negotiated authn=JWT
0927 21:27:18.022865 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:18.023638 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:18.025790 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:18.026318 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:18.026462 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:18.027126 (+ 664us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 21:27:18.097427 (+ 70301us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 21:27:18.097923 (+ 496us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41450: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 21:27:18.098394 5941 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41450: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 21:27:18.098397 5967 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:18.020766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33481 (local address 127.0.0.1:41450)
0927 21:27:18.021524 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:18.021547 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:18.021771 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:18.022320 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:18.022326 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:18.022342 (+ 16us) client_negotiation.cc:189] Negotiated authn=JWT
0927 21:27:18.022677 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:18.022688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:18.023820 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:18.023827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:18.025542 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:18.025561 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:18.026485 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:18.026552 (+ 67us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:18.026767 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:18.026821 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 21:27:18.097801 (+ 70980us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 21:27:18.098076 (+ 275us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:33481: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":407,"threads_started":1}
I20260927 21:27:18.099495 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 5872
I20260927 21:27:18.100805 5933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:18.204221 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 5872
2026-09-27T21:27:18Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (799 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (432 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1501 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:27:20 dist-test-slave-bxm0 krb5kdc[5983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:27:20 dist-test-slave-bxm0 krb5kdc[5983](info): set up 2 sockets
Sep 27 21:27:20 dist-test-slave-bxm0 krb5kdc[5983](info): commencing operation
krb5kdc: starting...
W20260927 21:27:21.238678 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:27:21 dist-test-slave-bxm0 krb5kdc[5983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:27:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:27:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:27:21.453702 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41429
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:21.784140 5996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:21.784525 5996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:21.784696 5996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:21.792249 5996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:27:21.792421 5996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:21.792654 5996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:21.792781 5996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:27:21.792899 5996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:27:21.803285 5996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:21.804719 5996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:21.806180 5996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:21.819093 6001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:21.819054 6002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:21.819016 6004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:21.820351 5996 server_base.cc:1034] running on GCE node
I20260927 21:27:21.821180 5996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:21.822924 5996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:21.824162 5996 hybrid_clock.cc:648] HybridClock initialized: now 1790544441824104 us; error 48 us; skew 500 ppm
I20260927 21:27:21.824504 5996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:21 dist-test-slave-bxm0 krb5kdc[5983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:21.832413 5996 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:27:21.834352 5996 webserver.cc:460] Webserver started at http://127.0.97.62:46433/ using document root <none> and password file <none>
I20260927 21:27:21.835100 5996 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:21.835213 5996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:21.835496 5996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:21.838817 5996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "2e7ddd1054704ab0842cf7c2f7d80ba1"
format_stamp: "Formatted at 2026-09-27 21:27:21 on dist-test-slave-bxm0"
server_key: "3e5cf97c65986386f20cbf7d06e0c4bc"
server_key_iv: "7b46f9bf215f9959e11a5bcbcc0b1b4a"
server_key_version: "encryptionkey@0"
I20260927 21:27:21.839622 5996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "2e7ddd1054704ab0842cf7c2f7d80ba1"
format_stamp: "Formatted at 2026-09-27 21:27:21 on dist-test-slave-bxm0"
server_key: "3e5cf97c65986386f20cbf7d06e0c4bc"
server_key_iv: "7b46f9bf215f9959e11a5bcbcc0b1b4a"
server_key_version: "encryptionkey@0"
I20260927 21:27:21.845314 5996 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:27:21.849139 6011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:21.850688 5996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:27:21.850915 5996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "2e7ddd1054704ab0842cf7c2f7d80ba1"
format_stamp: "Formatted at 2026-09-27 21:27:21 on dist-test-slave-bxm0"
server_key: "3e5cf97c65986386f20cbf7d06e0c4bc"
server_key_iv: "7b46f9bf215f9959e11a5bcbcc0b1b4a"
server_key_version: "encryptionkey@0"
I20260927 21:27:21.851146 5996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:21.871938 5996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:21.876248 5996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:21.876540 5996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:21.895263 5996 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41429
I20260927 21:27:21.895350 6062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41429 every 8 connection(s)
I20260927 21:27:21.896668 5996 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:27:21.900743 6063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:21.904479 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 5996
I20260927 21:27:21.904712 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:27:21.905578 388 external_mini_cluster.cc:1428] Setting key 1476d3564fb249acd82695572ccaee96
I20260927 21:27:21.914263 6063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e7ddd1054704ab0842cf7c2f7d80ba1: Bootstrap starting.
I20260927 21:27:21.917973 6063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e7ddd1054704ab0842cf7c2f7d80ba1: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:21.919085 6063 log.cc:824] T 00000000000000000000000000000000 P 2e7ddd1054704ab0842cf7c2f7d80ba1: Log is configured to *not* fsync() on all Append() calls
Sep 27 21:27:21 dist-test-slave-bxm0 krb5kdc[5983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:27:21.922771 6063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e7ddd1054704ab0842cf7c2f7d80ba1: No bootstrap required, opened a new log
I20260927 21:27:21.930481 6066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:21.907941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54574 (local address 127.0.97.62:41429)
0927 21:27:21.908715 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:21.908728 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:21.908769 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:27:21.910568 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:21.910593 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:21.911027 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:21.911474 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:21.912917 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:21.914287 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:21.915605 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:21.916305 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:21.916533 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:21.920912 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:21.920925 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:21.920945 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:21.920997 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:21.925272 (+ 4275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:21.926181 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:21.926192 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:21.926203 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:21.926449 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:21.926833 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:21.926838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:21.926841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:21.927666 (+ 825us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:21.927978 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:21.928381 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 21:27:21.928887 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":359,"threads_started":1}
I20260927 21:27:21.934229 6063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e7ddd1054704ab0842cf7c2f7d80ba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7ddd1054704ab0842cf7c2f7d80ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41429 } }
I20260927 21:27:21.934614 6063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e7ddd1054704ab0842cf7c2f7d80ba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:21.934700 6063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e7ddd1054704ab0842cf7c2f7d80ba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e7ddd1054704ab0842cf7c2f7d80ba1, State: Initialized, Role: FOLLOWER
I20260927 21:27:21.935317 6063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e7ddd1054704ab0842cf7c2f7d80ba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7ddd1054704ab0842cf7c2f7d80ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41429 } }
I20260927 21:27:21.935534 6063 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e7ddd1054704ab0842cf7c2f7d80ba1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:27:21.935642 6063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e7ddd1054704ab0842cf7c2f7d80ba1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:27:21.935787 6063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e7ddd1054704ab0842cf7c2f7d80ba1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:21.938385 6063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e7ddd1054704ab0842cf7c2f7d80ba1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7ddd1054704ab0842cf7c2f7d80ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41429 } }
I20260927 21:27:21.938892 6063 leader_election.cc:302] T 00000000000000000000000000000000 P 2e7ddd1054704ab0842cf7c2f7d80ba1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e7ddd1054704ab0842cf7c2f7d80ba1; no voters:
I20260927 21:27:21.939440 6063 leader_election.cc:288] T 00000000000000000000000000000000 P 2e7ddd1054704ab0842cf7c2f7d80ba1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:27:21.939589 6068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e7ddd1054704ab0842cf7c2f7d80ba1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:27:21.940315 6068 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e7ddd1054704ab0842cf7c2f7d80ba1 [term 1 LEADER]: Becoming Leader. State: Replica: 2e7ddd1054704ab0842cf7c2f7d80ba1, State: Running, Role: LEADER
I20260927 21:27:21.940840 6068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e7ddd1054704ab0842cf7c2f7d80ba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7ddd1054704ab0842cf7c2f7d80ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41429 } }
I20260927 21:27:21.941808 6063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e7ddd1054704ab0842cf7c2f7d80ba1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:27:21.944294 6069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e7ddd1054704ab0842cf7c2f7d80ba1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e7ddd1054704ab0842cf7c2f7d80ba1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7ddd1054704ab0842cf7c2f7d80ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41429 } } }
I20260927 21:27:21.944802 6069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e7ddd1054704ab0842cf7c2f7d80ba1 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:21.945870 6070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e7ddd1054704ab0842cf7c2f7d80ba1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e7ddd1054704ab0842cf7c2f7d80ba1. Latest consensus state: current_term: 1 leader_uuid: "2e7ddd1054704ab0842cf7c2f7d80ba1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7ddd1054704ab0842cf7c2f7d80ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41429 } } }
I20260927 21:27:21.946180 6074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:27:21.946271 6070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e7ddd1054704ab0842cf7c2f7d80ba1 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:21.950966 6074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:27:21.960448 6074 catalog_manager.cc:1347] Generated new cluster ID: 0731f34ebc734d559d5f4a17dc3cb3e8
I20260927 21:27:21.960582 6074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:27:21.987829 6074 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:27:21.989192 6074 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:27:21.998931 6074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e7ddd1054704ab0842cf7c2f7d80ba1: Generated new TSK 0
I20260927 21:27:21.999929 6074 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 21:27:22.059170 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41429
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:22.393355 6089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:22.393764 6089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:22.393950 6089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:22.401813 6089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:22.401949 6089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:22.402122 6089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:27:22.412421 6089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:22.413930 6089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:22.415307 6089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:22.428947 6094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:22.428923 6097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:22.428833 6095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:22.430967 6089 server_base.cc:1034] running on GCE node
I20260927 21:27:22.431685 6089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:22.432871 6089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:22.434110 6089 hybrid_clock.cc:648] HybridClock initialized: now 1790544442434036 us; error 64 us; skew 500 ppm
I20260927 21:27:22.434455 6089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:22 dist-test-slave-bxm0 krb5kdc[5983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:22.437855 6089 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 21:27:22.439432 6089 webserver.cc:460] Webserver started at http://127.0.97.1:33703/ using document root <none> and password file <none>
I20260927 21:27:22.440205 6089 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:22.440366 6089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:22.440701 6089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:22.444121 6089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-0/data/instance:
uuid: "b1629f4dfa674dfeafbcceea8acfe081"
format_stamp: "Formatted at 2026-09-27 21:27:22 on dist-test-slave-bxm0"
server_key: "d1186f1b9f63d88ee18e9c4fe5c709fa"
server_key_iv: "61588eb8240634aa9c7211d0e03fc60f"
server_key_version: "encryptionkey@0"
I20260927 21:27:22.445154 6089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b1629f4dfa674dfeafbcceea8acfe081"
format_stamp: "Formatted at 2026-09-27 21:27:22 on dist-test-slave-bxm0"
server_key: "d1186f1b9f63d88ee18e9c4fe5c709fa"
server_key_iv: "61588eb8240634aa9c7211d0e03fc60f"
server_key_version: "encryptionkey@0"
I20260927 21:27:22.450938 6089 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 21:27:22.454617 6104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:22.456149 6089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:27:22.456350 6089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "b1629f4dfa674dfeafbcceea8acfe081"
format_stamp: "Formatted at 2026-09-27 21:27:22 on dist-test-slave-bxm0"
server_key: "d1186f1b9f63d88ee18e9c4fe5c709fa"
server_key_iv: "61588eb8240634aa9c7211d0e03fc60f"
server_key_version: "encryptionkey@0"
I20260927 21:27:22.456578 6089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:22.486861 6089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:22.491101 6089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:22.491410 6089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:22.492376 6089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:27:22.493868 6089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:22.493968 6089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:22.494127 6089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:22.494197 6089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:22.527993 6089 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35931
I20260927 21:27:22.528054 6216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35931 every 8 connection(s)
I20260927 21:27:22.529402 6089 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:27:22.530218 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 6089
I20260927 21:27:22.530424 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-0/wal/instance
I20260927 21:27:22.531219 388 external_mini_cluster.cc:1428] Setting key fb324531b549f2a4cba4b665cfed23d0
Sep 27 21:27:22 dist-test-slave-bxm0 krb5kdc[5983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 21:27:22.554407 6219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:22.532235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57333 (local address 127.0.97.62:41429)
0927 21:27:22.532701 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:22.532706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:22.534083 (+ 1377us) server_negotiation.cc:407] Connection header received
0927 21:27:22.537008 (+ 2925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:22.537013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:22.537078 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:22.537174 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:22.539429 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:22.540186 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:22.542394 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:22.542850 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:22.542961 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:22.547704 (+ 4743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:22.547710 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:22.547713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:22.547759 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:22.550601 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:22.551701 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:22.551706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:22.551708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:22.551827 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:22.552394 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:22.552398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:22.552400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:22.552688 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:22.552822 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:22.553848 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0927 21:27:22.554092 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":252,"threads_started":1}
I20260927 21:27:22.555540 6220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:22.532969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41429 (local address 127.0.97.1:57333)
0927 21:27:22.533889 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:22.533949 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:22.536689 (+ 2740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:22.537477 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:22.537491 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:22.538320 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:22.539205 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:22.539234 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:22.540368 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:22.540381 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:22.542192 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:22.542231 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:22.543105 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:22.543114 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:22.543464 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:22.544240 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:22.544280 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:22.547374 (+ 3094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:22.550798 (+ 3424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:22.550812 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:22.550839 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:22.551551 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:22.551947 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:22.551953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:22.551956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:22.552229 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:22.552841 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:22.552850 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:22.553370 (+ 520us) client_negotiation.cc:769] Sending connection context
0927 21:27:22.553744 (+ 374us) client_negotiation.cc:240] Negotiation successful
0927 21:27:22.554348 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":382,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 21:27:22.557370 6217 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41429
I20260927 21:27:22.557740 6217 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:22.558503 6217 heartbeater.cc:511] Master 127.0.97.62:41429 requested a full tablet report, sending...
I20260927 21:27:22.560931 6028 ts_manager.cc:194] Registered new tserver with Master: b1629f4dfa674dfeafbcceea8acfe081 (127.0.97.1:35931)
I20260927 21:27:22.562636 6028 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 21:27:22.581446 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41429
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:22.915410 6223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:22.915763 6223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:22.915946 6223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:22.923584 6223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:22.923753 6223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:22.923947 6223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:27:22.934283 6223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:22.935669 6223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:22.937052 6223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:22.951479 6231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:22.951563 6229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:22.951604 6228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:22.952397 6223 server_base.cc:1034] running on GCE node
I20260927 21:27:22.953070 6223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:22.954170 6223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:22.955423 6223 hybrid_clock.cc:648] HybridClock initialized: now 1790544442955379 us; error 39 us; skew 500 ppm
I20260927 21:27:22.955839 6223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:22 dist-test-slave-bxm0 krb5kdc[5983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:22.959306 6223 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 21:27:22.960911 6223 webserver.cc:460] Webserver started at http://127.0.97.2:38275/ using document root <none> and password file <none>
I20260927 21:27:22.961721 6223 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:22.961867 6223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:22.962188 6223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:22.965097 6223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-1/data/instance:
uuid: "65d75b0c0c5248ddbee0768c0b9b12fe"
format_stamp: "Formatted at 2026-09-27 21:27:22 on dist-test-slave-bxm0"
server_key: "6fd3db01418d74672895ffe1b7c7996f"
server_key_iv: "86d4cec26fe41afe17573dab50c33c08"
server_key_version: "encryptionkey@0"
I20260927 21:27:22.965945 6223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-1/wal/instance:
uuid: "65d75b0c0c5248ddbee0768c0b9b12fe"
format_stamp: "Formatted at 2026-09-27 21:27:22 on dist-test-slave-bxm0"
server_key: "6fd3db01418d74672895ffe1b7c7996f"
server_key_iv: "86d4cec26fe41afe17573dab50c33c08"
server_key_version: "encryptionkey@0"
I20260927 21:27:22.971165 6223 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 21:27:22.974786 6238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:22.976282 6223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:27:22.976493 6223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "65d75b0c0c5248ddbee0768c0b9b12fe"
format_stamp: "Formatted at 2026-09-27 21:27:22 on dist-test-slave-bxm0"
server_key: "6fd3db01418d74672895ffe1b7c7996f"
server_key_iv: "86d4cec26fe41afe17573dab50c33c08"
server_key_version: "encryptionkey@0"
I20260927 21:27:22.976743 6223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:23.007508 6223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:23.012122 6223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:23.012427 6223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:23.013437 6223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:27:23.014964 6223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:23.015076 6223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:23.015228 6223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:23.015309 6223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:23.049115 6223 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36009
I20260927 21:27:23.049206 6350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36009 every 8 connection(s)
I20260927 21:27:23.050467 6223 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:27:23.052675 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 6223
I20260927 21:27:23.052863 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-1/wal/instance
I20260927 21:27:23.053438 388 external_mini_cluster.cc:1428] Setting key 45f9f12b6ba75e4d02bfd5cb9dedb345
Sep 27 21:27:23 dist-test-slave-bxm0 krb5kdc[5983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:27:23.074306 6219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:23.053238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60091 (local address 127.0.97.62:41429)
0927 21:27:23.053413 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:23.053418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:23.055030 (+ 1612us) server_negotiation.cc:407] Connection header received
0927 21:27:23.057061 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:23.057070 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:23.057158 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:23.057274 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:23.058798 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:23.059959 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:23.061771 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:23.062526 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:23.062671 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:23.066502 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:23.066511 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:23.066519 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:23.066564 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:23.070400 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:23.071367 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:23.071374 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:23.071379 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:23.071578 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:23.072126 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:23.072134 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:23.072138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:23.072587 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:23.072747 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:23.073643 (+ 896us) server_negotiation.cc:299] Negotiation successful
0927 21:27:23.073902 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 21:27:23.075320 6353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:23.053956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41429 (local address 127.0.97.2:60091)
0927 21:27:23.054860 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:23.054912 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:23.056788 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:23.057491 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:23.057499 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:23.058063 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:23.058632 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:23.058649 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:23.060108 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:23.060140 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:23.061576 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:23.061587 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:23.062722 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:23.062726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:23.062925 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:23.063604 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:23.063625 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:23.066189 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:23.070624 (+ 4435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:23.070632 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:23.070654 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:23.071179 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:23.071743 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:23.071747 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:23.071749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:23.071977 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:23.072745 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:23.072752 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:23.073179 (+ 427us) client_negotiation.cc:769] Sending connection context
0927 21:27:23.073433 (+ 254us) client_negotiation.cc:240] Negotiation successful
0927 21:27:23.073984 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":384,"threads_started":1}
I20260927 21:27:23.076897 6351 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41429
I20260927 21:27:23.077174 6351 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 21:27:23.078231 6351 heartbeater.cc:511] Master 127.0.97.62:41429 requested a full tablet report, sending...
I20260927 21:27:23.079905 6028 ts_manager.cc:194] Registered new tserver with Master: 65d75b0c0c5248ddbee0768c0b9b12fe (127.0.97.2:36009)
I20260927 21:27:23.080803 6028 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 21:27:23.116771 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41429
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:23.454035 6356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:23.454384 6356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:23.454542 6356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:23.462203 6356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:23.462322 6356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:23.462500 6356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:27:23.472615 6356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:23.474190 6356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:23.475636 6356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:23.489727 6361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:23.489725 6362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:23.489725 6364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:23.490720 6356 server_base.cc:1034] running on GCE node
I20260927 21:27:23.491323 6356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:23.492410 6356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:23.493762 6356 hybrid_clock.cc:648] HybridClock initialized: now 1790544443493678 us; error 67 us; skew 500 ppm
I20260927 21:27:23.494174 6356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:23 dist-test-slave-bxm0 krb5kdc[5983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:23.498066 6356 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 21:27:23.499666 6356 webserver.cc:460] Webserver started at http://127.0.97.3:35883/ using document root <none> and password file <none>
I20260927 21:27:23.500401 6356 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:23.500553 6356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:23.500874 6356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:23.504053 6356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-2/data/instance:
uuid: "5a3e98270ee9405cbe5882613c9dc3ef"
format_stamp: "Formatted at 2026-09-27 21:27:23 on dist-test-slave-bxm0"
server_key: "0c43e3837e6a9d01e9b7d3a78a49270f"
server_key_iv: "f4ff9ebf71cd04f9badd401dc3c77a80"
server_key_version: "encryptionkey@0"
I20260927 21:27:23.504855 6356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5a3e98270ee9405cbe5882613c9dc3ef"
format_stamp: "Formatted at 2026-09-27 21:27:23 on dist-test-slave-bxm0"
server_key: "0c43e3837e6a9d01e9b7d3a78a49270f"
server_key_iv: "f4ff9ebf71cd04f9badd401dc3c77a80"
server_key_version: "encryptionkey@0"
I20260927 21:27:23.510389 6356 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 21:27:23.514045 6371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:23.515743 6356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 21:27:23.515962 6356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "5a3e98270ee9405cbe5882613c9dc3ef"
format_stamp: "Formatted at 2026-09-27 21:27:23 on dist-test-slave-bxm0"
server_key: "0c43e3837e6a9d01e9b7d3a78a49270f"
server_key_iv: "f4ff9ebf71cd04f9badd401dc3c77a80"
server_key_version: "encryptionkey@0"
I20260927 21:27:23.516193 6356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:23.537297 6356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:23.542055 6356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:23.542351 6356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:23.543452 6356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:27:23.544806 6356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:23.544916 6356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:23.545076 6356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:23.545151 6356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:23.566570 6217 heartbeater.cc:503] Master 127.0.97.62:41429 was elected leader, sending a full tablet report...
I20260927 21:27:23.580686 6356 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42757
I20260927 21:27:23.580699 6483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42757 every 8 connection(s)
I20260927 21:27:23.582207 6356 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:27:23.587998 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 6356
I20260927 21:27:23.588198 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/minicluster-data/ts-2/wal/instance
I20260927 21:27:23.588788 388 external_mini_cluster.cc:1428] Setting key 2669c9a95440b72bc39df98da0630d25
Sep 27 21:27:23 dist-test-slave-bxm0 krb5kdc[5983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:27:23.604269 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:23.584752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54677 (local address 127.0.97.62:41429)
0927 21:27:23.585195 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:23.585202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:23.586220 (+ 1018us) server_negotiation.cc:407] Connection header received
0927 21:27:23.588213 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:23.588226 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:23.588332 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:23.588429 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:23.590431 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:23.591304 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:23.593078 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:23.593456 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:23.593555 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:23.597283 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:23.597291 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:23.597296 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:23.597338 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:23.600529 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:23.601364 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:23.601369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:23.601371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:23.601515 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:23.602056 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:23.602060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:23.602063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:23.602340 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:23.602543 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:23.603529 (+ 986us) server_negotiation.cc:299] Negotiation successful
0927 21:27:23.603741 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":206,"threads_started":1}
I20260927 21:27:23.605407 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:23.585320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41429 (local address 127.0.97.3:54677)
0927 21:27:23.586062 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:23.586105 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:23.587867 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:23.588711 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:23.588723 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:23.589430 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:23.590226 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:23.590246 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:23.591449 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:23.591454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:23.592935 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:23.592948 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:23.593753 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:23.593760 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:23.593981 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:23.594643 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:23.594670 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:23.597034 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:23.600706 (+ 3672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:23.600714 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:23.600730 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:23.601210 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:23.601674 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:23.601683 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:23.601685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:23.601904 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:23.602504 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:23.602514 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:23.602969 (+ 455us) client_negotiation.cc:769] Sending connection context
0927 21:27:23.603308 (+ 339us) client_negotiation.cc:240] Negotiation successful
0927 21:27:23.603947 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":330,"threads_started":1}
I20260927 21:27:23.606849 6484 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41429
I20260927 21:27:23.607180 6484 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:23.607990 6484 heartbeater.cc:511] Master 127.0.97.62:41429 requested a full tablet report, sending...
I20260927 21:27:23.609503 6028 ts_manager.cc:194] Registered new tserver with Master: 5a3e98270ee9405cbe5882613c9dc3ef (127.0.97.3:42757)
I20260927 21:27:23.610330 6028 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54677
I20260927 21:27:23.617098 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/test-admin.keytab.
Sep 27 21:27:23 dist-test-slave-bxm0 krb5kdc[5983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:23.645454 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 21:27:23.651288 6492 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:23.648661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58708 (local address 127.0.97.1:35931)
0927 21:27:23.649276 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:23.649284 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:23.649313 (+ 29us) server_negotiation.cc:407] Connection header received
0927 21:27:23.649479 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:23.649486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:23.649791 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:23.649936 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:23.650893 (+ 957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58708: BlockingRecv error: recv got EOF from 127.0.0.1:58708 (error 108)
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":327,"threads_started":1}
W20260927 21:27:23.651306 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/krb5cc' of type FILE
Sep 27 21:27:23 dist-test-slave-bxm0 krb5kdc[5983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:23.653156 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 21:27:23 dist-test-slave-bxm0 krb5kdc[5983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:23.654502 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 21:27:23 dist-test-slave-bxm0 krb5kdc[5983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 21:27:23.670742 6492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:23.656467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58722 (local address 127.0.97.1:35931)
0927 21:27:23.656664 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:23.656670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:23.656777 (+ 107us) server_negotiation.cc:407] Connection header received
0927 21:27:23.657035 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:23.657041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:23.657103 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:23.657210 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:23.658258 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:23.659224 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:23.660571 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:23.661214 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:23.661347 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:23.665029 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:23.665041 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:23.665056 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:23.665090 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:23.667856 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:23.668602 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:23.668610 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:23.668612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:23.668735 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:23.669122 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:23.669128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:23.669131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:23.669511 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:23.669825 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:23.670167 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 21:27:23.670440 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 21:27:23.676705 6492 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:23.675036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58734 (local address 127.0.97.1:35931)
0927 21:27:23.675248 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:23.675255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:23.675446 (+ 191us) server_negotiation.cc:407] Connection header received
0927 21:27:23.675547 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:23.675551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:23.675620 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:23.675704 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:23.676452 (+ 748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58734: BlockingRecv error: recv got EOF from 127.0.0.1:58734 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260927 21:27:23.676777 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/krb5cc' of type FILE
Sep 27 21:27:23 dist-test-slave-bxm0 krb5kdc[5983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:23.678447 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 21:27:23 dist-test-slave-bxm0 krb5kdc[5983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:23.679742 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 21:27:23 dist-test-slave-bxm0 krb5kdc[5983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 21:27:23.694221 6492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:23.681125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58742 (local address 127.0.97.1:35931)
0927 21:27:23.681330 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:23.681351 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:23.681659 (+ 308us) server_negotiation.cc:407] Connection header received
0927 21:27:23.681865 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:23.681875 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:23.681941 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:23.682049 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:23.682955 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:23.683668 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:23.684854 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:23.685267 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:23.685408 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:23.689044 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:23.689050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:23.689053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:23.689096 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:23.691687 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:23.692351 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:23.692363 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:23.692366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:23.692490 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:23.692883 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:23.692899 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:23.692908 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:23.693165 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:23.693431 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:23.693731 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 21:27:23.693943 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 21:27:23.698768 6492 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:23.696860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58750 (local address 127.0.97.1:35931)
0927 21:27:23.697106 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:23.697113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:23.697351 (+ 238us) server_negotiation.cc:407] Connection header received
0927 21:27:23.697548 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:23.697553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:23.697694 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:23.697804 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:23.698503 (+ 699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58750: BlockingRecv error: recv got EOF from 127.0.0.1:58750 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
W20260927 21:27:23.698935 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790544390748963-388-0/krb5kdc/krb5cc' of type FILE
Sep 27 21:27:23 dist-test-slave-bxm0 krb5kdc[5983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:23.700510 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 21:27:23 dist-test-slave-bxm0 krb5kdc[5983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:23.701838 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 21:27:23 dist-test-slave-bxm0 krb5kdc[5983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 21:27:23.716786 6492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:23.703199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58754 (local address 127.0.97.1:35931)
0927 21:27:23.703419 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:23.703427 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:23.703689 (+ 262us) server_negotiation.cc:407] Connection header received
0927 21:27:23.703885 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:23.703890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:23.703954 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:23.704066 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:23.704950 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:23.705744 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:23.707123 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:23.707569 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:23.707715 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:23.711502 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:23.711507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:23.711511 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:23.711546 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:23.714151 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:23.714834 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:23.714839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:23.714843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:23.715008 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:23.715373 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:23.715378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:23.715382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:23.715657 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:23.715991 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:23.716247 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 21:27:23.716467 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 21:27:23.718564 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 6089
I20260927 21:27:23.730736 6492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:23.719363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58762 (local address 127.0.97.1:35931)
0927 21:27:23.719554 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:23.719558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:23.719887 (+ 329us) server_negotiation.cc:407] Connection header received
0927 21:27:23.720042 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:23.720048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:23.720120 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:23.720219 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:23.721162 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:23.721930 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:23.723086 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:23.723527 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:23.723664 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:23.725819 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:23.725824 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:23.725827 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:23.725860 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:23.728168 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:23.728810 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:23.728815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:23.728824 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:23.728958 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:23.729320 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:23.729326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:23.729329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:23.729597 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:23.729895 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:23.730208 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 21:27:23.730412 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 21:27:23.731381 6212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:23.815706 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 6089
I20260927 21:27:23.832505 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 6223
Sep 27 21:27:23 dist-test-slave-bxm0 krb5kdc[5983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 21:27:23.848321 6505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:23.833432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58830 (local address 127.0.97.2:36009)
0927 21:27:23.834019 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:23.834026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:23.834050 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:27:23.834109 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:23.834115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:23.834323 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:23.834502 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:23.835464 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:23.836459 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:23.837752 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:23.838298 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:23.838440 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:23.842395 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:23.842408 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:23.842424 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:23.842460 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:23.845344 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:23.846153 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:23.846161 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:23.846163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:23.846289 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:23.846697 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:23.846701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:23.846703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:23.847058 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:23.847363 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:23.847721 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 21:27:23.847988 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":317,"threads_started":1}
I20260927 21:27:23.849680 6346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:23.932616 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 6223
I20260927 21:27:23.948382 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 6356
Sep 27 21:27:23 dist-test-slave-bxm0 krb5kdc[5983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 21:27:23.964965 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:23.949403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.0.97.3:42757)
0927 21:27:23.950081 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:23.950088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:23.950115 (+ 27us) server_negotiation.cc:407] Connection header received
0927 21:27:23.950172 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:23.950178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:23.950381 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:23.950554 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:23.951593 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:23.952604 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:23.953939 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:23.954498 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:23.954638 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:23.959104 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:23.959125 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:23.959146 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:23.959190 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:23.962102 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:23.962861 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:23.962873 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:23.962876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:23.963006 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:23.963391 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:23.963397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:23.963400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:23.963766 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:23.964095 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:23.964383 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 21:27:23.964649 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":381,"threads_started":1}
I20260927 21:27:23.966399 6479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:24.049669 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 6356
I20260927 21:27:24.067363 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 5996
I20260927 21:27:24.068519 6058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:24.172631 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 5996
2026-09-27T21:27:24Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4039 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:27:24 dist-test-slave-bxm0 krb5kdc[6512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:27:24 dist-test-slave-bxm0 krb5kdc[6512](info): set up 2 sockets
Sep 27 21:27:24 dist-test-slave-bxm0 krb5kdc[6512](info): commencing operation
krb5kdc: starting...
W20260927 21:27:25.270538 388 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:27:25 dist-test-slave-bxm0 krb5kdc[6512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:27:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:27:25Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260927 21:27:25.488960 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39737
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:25.819839 6525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:25.820137 6525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:25.820225 6525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:25.827628 6525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:27:25.827754 6525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:25.827800 6525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:25.827857 6525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:27:25.827903 6525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:27:25.837941 6525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:25.839300 6525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:25.840547 6525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:25.852195 6530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:25.852103 6533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:25.852134 6531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:25.853243 6525 server_base.cc:1034] running on GCE node
I20260927 21:27:25.854121 6525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:25.855484 6525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:25.856700 6525 hybrid_clock.cc:648] HybridClock initialized: now 1790544445856660 us; error 33 us; skew 500 ppm
I20260927 21:27:25.857076 6525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:25 dist-test-slave-bxm0 krb5kdc[6512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544445, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:25.860198 6525 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260927 21:27:25.861737 6525 webserver.cc:460] Webserver started at http://127.0.97.62:33983/ using document root <none> and password file <none>
I20260927 21:27:25.862398 6525 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:25.862496 6525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:25.862733 6525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:25.865787 6525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "a978f7733b8c466f9f9ef4d1acefb6d2"
format_stamp: "Formatted at 2026-09-27 21:27:25 on dist-test-slave-bxm0"
server_key: "109733f90a81c828fd688d55d218ceb0"
server_key_iv: "5c83f7949578fa8c431a7505a459402c"
server_key_version: "encryptionkey@0"
I20260927 21:27:25.866540 6525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "a978f7733b8c466f9f9ef4d1acefb6d2"
format_stamp: "Formatted at 2026-09-27 21:27:25 on dist-test-slave-bxm0"
server_key: "109733f90a81c828fd688d55d218ceb0"
server_key_iv: "5c83f7949578fa8c431a7505a459402c"
server_key_version: "encryptionkey@0"
I20260927 21:27:25.871744 6525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:27:25.875324 6540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:25.876840 6525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 21:27:25.877058 6525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "a978f7733b8c466f9f9ef4d1acefb6d2"
format_stamp: "Formatted at 2026-09-27 21:27:25 on dist-test-slave-bxm0"
server_key: "109733f90a81c828fd688d55d218ceb0"
server_key_iv: "5c83f7949578fa8c431a7505a459402c"
server_key_version: "encryptionkey@0"
I20260927 21:27:25.877269 6525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:25.891651 6525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:25.896265 6525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:25.896582 6525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:25.915218 6525 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39737
I20260927 21:27:25.915256 6591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39737 every 8 connection(s)
I20260927 21:27:25.916661 6525 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:27:25.920159 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 6525
I20260927 21:27:25.920372 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:27:25.920960 6592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:25.921211 388 external_mini_cluster.cc:1428] Setting key 3abd19d320abe202d742a77ff832e49a
I20260927 21:27:25.936317 6592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2: Bootstrap starting.
Sep 27 21:27:25 dist-test-slave-bxm0 krb5kdc[6512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 21:27:25.939878 6592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:25.940994 6592 log.cc:824] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2: Log is configured to *not* fsync() on all Append() calls
I20260927 21:27:25.943925 6592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2: No bootstrap required, opened a new log
I20260927 21:27:25.945001 6595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:25.923300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53404 (local address 127.0.97.62:39737)
0927 21:27:25.924132 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:25.924143 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:25.924188 (+ 45us) server_negotiation.cc:407] Connection header received
0927 21:27:25.926070 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:25.926096 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:25.926528 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:25.926891 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:25.928694 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:25.930106 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:25.931907 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:25.932704 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:25.932854 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:25.937673 (+ 4819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:25.937684 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:25.937711 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:25.937767 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:25.940874 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:25.941693 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:25.941700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:25.941706 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:25.941871 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:25.942269 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:25.942275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:25.942278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:25.942801 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:25.943099 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:25.943541 (+ 442us) server_negotiation.cc:299] Negotiation successful
0927 21:27:25.943955 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":388,"threads_started":1}
I20260927 21:27:25.951586 6592 raft_consensus.cc:348] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } }
I20260927 21:27:25.951985 6592 raft_consensus.cc:374] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:25.952097 6592 raft_consensus.cc:729] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a978f7733b8c466f9f9ef4d1acefb6d2, State: Initialized, Role: FOLLOWER
I20260927 21:27:25.952725 6592 consensus_queue.cc:260] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } }
I20260927 21:27:25.952967 6592 raft_consensus.cc:388] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:27:25.953114 6592 raft_consensus.cc:482] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:27:25.953286 6592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:25.955685 6592 raft_consensus.cc:504] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } }
I20260927 21:27:25.956254 6592 leader_election.cc:302] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a978f7733b8c466f9f9ef4d1acefb6d2; no voters:
I20260927 21:27:25.956918 6592 leader_election.cc:288] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:27:25.957037 6597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:27:25.957751 6597 raft_consensus.cc:686] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [term 1 LEADER]: Becoming Leader. State: Replica: a978f7733b8c466f9f9ef4d1acefb6d2, State: Running, Role: LEADER
I20260927 21:27:25.958271 6597 consensus_queue.cc:237] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } }
I20260927 21:27:25.959043 6592 sys_catalog.cc:564] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:27:25.960992 6599 sys_catalog.cc:455] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a978f7733b8c466f9f9ef4d1acefb6d2. Latest consensus state: current_term: 1 leader_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } }
I20260927 21:27:25.961395 6599 sys_catalog.cc:458] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:25.962813 6598 sys_catalog.cc:455] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } }
I20260927 21:27:25.963860 6598 sys_catalog.cc:458] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:25.965353 6603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:27:25.971691 6603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:27:25.979867 6603 catalog_manager.cc:1347] Generated new cluster ID: 700f24069d884b139152c6fa230b5dd4
I20260927 21:27:25.979997 6603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:27:25.993028 6603 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:27:25.994532 6603 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:27:26.007112 6603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2: Generated new TSK 0
I20260927 21:27:26.008186 6603 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260927 21:27:26.077323 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39737
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:26.421785 6618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:26.422094 6618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:26.422273 6618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:26.429546 6618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:26.429736 6618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:26.429939 6618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:27:26.439934 6618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:26.441257 6618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:26.442631 6618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:26.456015 6623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:26.456166 6626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:26.456306 6624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:26.460568 6618 server_base.cc:1034] running on GCE node
I20260927 21:27:26.461196 6618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:26.462244 6618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:26.463465 6618 hybrid_clock.cc:648] HybridClock initialized: now 1790544446463416 us; error 38 us; skew 500 ppm
I20260927 21:27:26.463773 6618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:26 dist-test-slave-bxm0 krb5kdc[6512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544446, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:26.467185 6618 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 21:27:26.468756 6618 webserver.cc:460] Webserver started at http://127.0.97.1:34415/ using document root <none> and password file <none>
I20260927 21:27:26.469499 6618 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:26.469652 6618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:26.469997 6618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:26.473150 6618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/data/instance:
uuid: "7f60711999224fc782e173cc4e4af402"
format_stamp: "Formatted at 2026-09-27 21:27:26 on dist-test-slave-bxm0"
server_key: "3b47232ca91e68c3052a7f98f34b3a30"
server_key_iv: "73d435b7075606164cd8a2c68abec9ea"
server_key_version: "encryptionkey@0"
I20260927 21:27:26.474013 6618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7f60711999224fc782e173cc4e4af402"
format_stamp: "Formatted at 2026-09-27 21:27:26 on dist-test-slave-bxm0"
server_key: "3b47232ca91e68c3052a7f98f34b3a30"
server_key_iv: "73d435b7075606164cd8a2c68abec9ea"
server_key_version: "encryptionkey@0"
I20260927 21:27:26.479496 6618 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 21:27:26.483451 6633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:26.484923 6618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:27:26.485111 6618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "7f60711999224fc782e173cc4e4af402"
format_stamp: "Formatted at 2026-09-27 21:27:26 on dist-test-slave-bxm0"
server_key: "3b47232ca91e68c3052a7f98f34b3a30"
server_key_iv: "73d435b7075606164cd8a2c68abec9ea"
server_key_version: "encryptionkey@0"
I20260927 21:27:26.485337 6618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:26.504449 6618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:26.508941 6618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:26.509254 6618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:26.510360 6618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:27:26.511775 6618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:26.511845 6618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:26.511932 6618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:26.511986 6618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:26.545789 6618 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33945
I20260927 21:27:26.545886 6745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33945 every 8 connection(s)
I20260927 21:27:26.547133 6618 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:27:26.551805 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 6618
I20260927 21:27:26.552003 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/wal/instance
I20260927 21:27:26.552655 388 external_mini_cluster.cc:1428] Setting key 116d0906833442e92f0055b2d961101a
Sep 27 21:27:26 dist-test-slave-bxm0 krb5kdc[6512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544446, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 21:27:26.569363 6748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:26.549781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58929 (local address 127.0.97.62:39737)
0927 21:27:26.550263 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:26.550270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:26.551264 (+ 994us) server_negotiation.cc:407] Connection header received
0927 21:27:26.553230 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:26.553236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:26.553340 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:26.553476 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:26.555223 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:26.556470 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:26.558317 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:26.559041 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:26.559196 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:26.563462 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:26.563467 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:26.563470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:26.563503 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:26.566294 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:26.567075 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:26.567081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:26.567085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:26.567204 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:26.567630 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:26.567634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:26.567638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:26.567900 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:26.568047 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:26.568768 (+ 721us) server_negotiation.cc:299] Negotiation successful
0927 21:27:26.568991 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":243,"threads_started":1}
I20260927 21:27:26.570396 6749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:26.550379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.1:58929)
0927 21:27:26.551094 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:26.551133 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:26.552952 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:26.553755 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:26.553765 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:26.554364 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:26.555039 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:26.555057 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:26.556667 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:26.556674 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:26.558145 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:26.558169 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:26.559343 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:26.559349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:26.559562 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:26.560349 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:26.560390 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:26.563196 (+ 2806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:26.566467 (+ 3271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:26.566493 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:26.566517 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:26.566941 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:26.567315 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:26.567321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:26.567324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:26.567512 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:26.568026 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:26.568034 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:26.568391 (+ 357us) client_negotiation.cc:769] Sending connection context
0927 21:27:26.568663 (+ 272us) client_negotiation.cc:240] Negotiation successful
0927 21:27:26.569213 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":287,"threads_started":1}
I20260927 21:27:26.572140 6746 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39737
I20260927 21:27:26.572455 6746 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 21:27:26.573242 6746 heartbeater.cc:511] Master 127.0.97.62:39737 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260927 21:27:26.575568 6557 ts_manager.cc:194] Registered new tserver with Master: 7f60711999224fc782e173cc4e4af402 (127.0.97.1:33945)
I20260927 21:27:26.577018 6557 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260927 21:27:26.600983 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39737
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:26.936820 6752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:26.937141 6752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:26.937304 6752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:26.944837 6752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:26.944967 6752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:26.945154 6752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:27:26.955158 6752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:26.956487 6752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:26.957839 6752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:26.971158 6757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:26.971228 6758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:26.971765 6760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:26.972429 6752 server_base.cc:1034] running on GCE node
I20260927 21:27:26.973117 6752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:26.974265 6752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:26.975484 6752 hybrid_clock.cc:648] HybridClock initialized: now 1790544446975450 us; error 37 us; skew 500 ppm
I20260927 21:27:26.975826 6752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:26 dist-test-slave-bxm0 krb5kdc[6512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544446, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:26.984328 6752 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 21:27:26.985954 6752 webserver.cc:460] Webserver started at http://127.0.97.2:36323/ using document root <none> and password file <none>
I20260927 21:27:26.986650 6752 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:26.986755 6752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:26.987022 6752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:26.990075 6752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/data/instance:
uuid: "4ff9eaae578a43f0aa15395e349d3117"
format_stamp: "Formatted at 2026-09-27 21:27:26 on dist-test-slave-bxm0"
server_key: "4c1d4793c366155f0e57d927739c8880"
server_key_iv: "3761cc8eb958cd127fb399badf285fc9"
server_key_version: "encryptionkey@0"
I20260927 21:27:26.990833 6752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4ff9eaae578a43f0aa15395e349d3117"
format_stamp: "Formatted at 2026-09-27 21:27:26 on dist-test-slave-bxm0"
server_key: "4c1d4793c366155f0e57d927739c8880"
server_key_iv: "3761cc8eb958cd127fb399badf285fc9"
server_key_version: "encryptionkey@0"
I20260927 21:27:26.996263 6752 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:27:27.000054 6767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:27.001950 6752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:27:27.002132 6752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "4ff9eaae578a43f0aa15395e349d3117"
format_stamp: "Formatted at 2026-09-27 21:27:26 on dist-test-slave-bxm0"
server_key: "4c1d4793c366155f0e57d927739c8880"
server_key_iv: "3761cc8eb958cd127fb399badf285fc9"
server_key_version: "encryptionkey@0"
I20260927 21:27:27.002382 6752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:27.024477 6752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:27.028753 6752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:27.029011 6752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:27.029997 6752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:27:27.031292 6752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:27.031365 6752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:27.031452 6752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:27.031497 6752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:27.065834 6752 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38203
I20260927 21:27:27.065900 6879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38203 every 8 connection(s)
I20260927 21:27:27.067209 6752 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:27:27.073037 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 6752
I20260927 21:27:27.073228 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/wal/instance
I20260927 21:27:27.073750 388 external_mini_cluster.cc:1428] Setting key 66376db9e94c3f75247df30d59b6a2aa
Sep 27 21:27:27 dist-test-slave-bxm0 krb5kdc[6512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544446, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 21:27:27.090499 6882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:27.069879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37665 (local address 127.0.97.62:39737)
0927 21:27:27.070359 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:27.070364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:27.071437 (+ 1073us) server_negotiation.cc:407] Connection header received
0927 21:27:27.073414 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:27.073420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:27.073510 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:27.073676 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:27.075642 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:27.076760 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:27.078464 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:27.079211 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:27.079359 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:27.083872 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:27.083878 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:27.083881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:27.083915 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:27.086607 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:27.087606 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:27.087611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:27.087613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:27.087752 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:27.088309 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:27.088313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:27.088316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:27.088722 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:27.088982 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:27.090050 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0927 21:27:27.090233 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":251,"threads_started":1}
I20260927 21:27:27.092008 6883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:27.070323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.2:37665)
0927 21:27:27.071265 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:27.071316 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:27.073080 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:27.073965 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:27.073974 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:27.074545 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:27.075179 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:27.075221 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:27.076932 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:27.076937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:27.078319 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:27.078331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:27.079490 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:27.079495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:27.079721 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:27.080539 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:27.080569 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:27.083551 (+ 2982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:27.086761 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:27.086774 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:27.086795 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:27.087450 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:27.087885 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:27.087893 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:27.087899 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:27.088152 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:27.088986 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:27.088997 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:27.089482 (+ 485us) client_negotiation.cc:769] Sending connection context
0927 21:27:27.089827 (+ 345us) client_negotiation.cc:240] Negotiation successful
0927 21:27:27.090438 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":295,"threads_started":1}
I20260927 21:27:27.093710 6880 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39737
I20260927 21:27:27.094043 6880 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:27.094772 6880 heartbeater.cc:511] Master 127.0.97.62:39737 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 21:27:27.096346 6557 ts_manager.cc:194] Registered new tserver with Master: 4ff9eaae578a43f0aa15395e349d3117 (127.0.97.2:38203)
I20260927 21:27:27.097232 6557 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260927 21:27:27.125061 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39737
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:27.463371 6886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:27.463704 6886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:27.463932 6886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:27.471670 6886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:27.471815 6886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:27.472056 6886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:27:27.482542 6886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:27.483965 6886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:27.485345 6886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:27.499337 6891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:27.499377 6894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:27.499483 6892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:27.500427 6886 server_base.cc:1034] running on GCE node
I20260927 21:27:27.501101 6886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:27.502204 6886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:27.503453 6886 hybrid_clock.cc:648] HybridClock initialized: now 1790544447503402 us; error 46 us; skew 500 ppm
I20260927 21:27:27.503783 6886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:27 dist-test-slave-bxm0 krb5kdc[6512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544447, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:27.507207 6886 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 21:27:27.508771 6886 webserver.cc:460] Webserver started at http://127.0.97.3:36815/ using document root <none> and password file <none>
I20260927 21:27:27.509487 6886 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:27.509699 6886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:27.510018 6886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:27.512996 6886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/data/instance:
uuid: "277d8b80cca4431c9917b2ea4570c077"
format_stamp: "Formatted at 2026-09-27 21:27:27 on dist-test-slave-bxm0"
server_key: "cca73652a7ce2d2d9e10a0037aca61f5"
server_key_iv: "73f5d0b868e78ec3658b98e853004577"
server_key_version: "encryptionkey@0"
I20260927 21:27:27.513860 6886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/wal/instance:
uuid: "277d8b80cca4431c9917b2ea4570c077"
format_stamp: "Formatted at 2026-09-27 21:27:27 on dist-test-slave-bxm0"
server_key: "cca73652a7ce2d2d9e10a0037aca61f5"
server_key_iv: "73f5d0b868e78ec3658b98e853004577"
server_key_version: "encryptionkey@0"
I20260927 21:27:27.519106 6886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 21:27:27.522702 6901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:27.524086 6886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:27:27.524267 6886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "277d8b80cca4431c9917b2ea4570c077"
format_stamp: "Formatted at 2026-09-27 21:27:27 on dist-test-slave-bxm0"
server_key: "cca73652a7ce2d2d9e10a0037aca61f5"
server_key_iv: "73f5d0b868e78ec3658b98e853004577"
server_key_version: "encryptionkey@0"
I20260927 21:27:27.524472 6886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:27.539762 6886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:27.544641 6886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:27.545040 6886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:27.546263 6886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:27:27.547830 6886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:27.547941 6886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:27.548102 6886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:27.548184 6886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:27.580209 6746 heartbeater.cc:503] Master 127.0.97.62:39737 was elected leader, sending a full tablet report...
I20260927 21:27:27.583091 6886 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33731
I20260927 21:27:27.583135 7013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33731 every 8 connection(s)
I20260927 21:27:27.584348 6886 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:27:27.586018 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 6886
I20260927 21:27:27.586190 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/wal/instance
I20260927 21:27:27.586820 388 external_mini_cluster.cc:1428] Setting key e68d1c788de40707b43a8a2950e04bdf
Sep 27 21:27:27 dist-test-slave-bxm0 krb5kdc[6512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544447, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 21:27:27.605127 6882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:27.587196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35537 (local address 127.0.97.62:39737)
0927 21:27:27.587408 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:27.587414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:27.588910 (+ 1496us) server_negotiation.cc:407] Connection header received
0927 21:27:27.590899 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:27.590903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:27.590971 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:27.591057 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:27.592465 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:27.593195 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:27.594795 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:27.595234 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:27.595347 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:27.599018 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:27.599022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:27.599026 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:27.599064 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:27.602056 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:27.602801 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:27.602813 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:27.602816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:27.602936 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:27.603373 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:27.603378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:27.603381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:27.603656 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:27.603815 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:27.604549 (+ 734us) server_negotiation.cc:299] Negotiation successful
0927 21:27:27.604776 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 21:27:27.606228 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:27.587846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.3:35537)
0927 21:27:27.588752 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:27.588798 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:27.590632 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:27.591237 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:27.591246 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:27.591742 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:27.592312 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:27.592329 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:27.593341 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:27.593345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:27.594643 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:27.594655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:27.595460 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:27.595466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:27.595684 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:27.596449 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:27.596472 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:27.598781 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:27.602208 (+ 3427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:27.602225 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:27.602240 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:27.602664 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:27.603049 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:27.603053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:27.603059 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:27.603246 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:27.603786 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:27.603795 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:27.604167 (+ 372us) client_negotiation.cc:769] Sending connection context
0927 21:27:27.604451 (+ 284us) client_negotiation.cc:240] Negotiation successful
0927 21:27:27.604997 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":397,"threads_started":1}
I20260927 21:27:27.607622 7014 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39737
I20260927 21:27:27.607934 7014 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:27.608783 7014 heartbeater.cc:511] Master 127.0.97.62:39737 requested a full tablet report, sending...
I20260927 21:27:27.610270 6557 ts_manager.cc:194] Registered new tserver with Master: 277d8b80cca4431c9917b2ea4570c077 (127.0.97.3:33731)
I20260927 21:27:27.611033 6557 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35537
I20260927 21:27:27.615082 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260927 21:27:27.669008 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34259
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39737,127.0.97.61:34259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:28.018779 7019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:28.019084 7019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:28.019170 7019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:28.026582 7019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:27:28.026700 7019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:28.026746 7019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:28.026803 7019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:27:28.026847 7019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:27:28.036720 7019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39737,127.0.97.61:34259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
W20260927 21:27:28.037057 7019 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:39737,127.0.97.61:34259'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 21:27:28.038089 7019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:28.039363 7019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:28.051007 7027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:28.051033 7025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:28.051069 7024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:28.052003 7019 server_base.cc:1034] running on GCE node
I20260927 21:27:28.052766 7019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:28.054250 7019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:28.055470 7019 hybrid_clock.cc:648] HybridClock initialized: now 1790544448055402 us; error 47 us; skew 500 ppm
I20260927 21:27:28.055748 7019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:28 dist-test-slave-bxm0 krb5kdc[6512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544448, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:28.059043 7019 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260927 21:27:28.060618 7019 webserver.cc:460] Webserver started at http://127.0.97.61:35595/ using document root <none> and password file <none>
I20260927 21:27:28.061305 7019 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:28.061398 7019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:28.061810 7019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:28.064719 7019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-1/data/instance:
uuid: "9f5af1173bb24003b4cac6a9beac8b91"
format_stamp: "Formatted at 2026-09-27 21:27:28 on dist-test-slave-bxm0"
server_key: "2485dde8723650353c1cca714d7bbad5"
server_key_iv: "891b6f69b84633ca03b4577cc639baad"
server_key_version: "encryptionkey@0"
I20260927 21:27:28.065559 7019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-1/wal/instance:
uuid: "9f5af1173bb24003b4cac6a9beac8b91"
format_stamp: "Formatted at 2026-09-27 21:27:28 on dist-test-slave-bxm0"
server_key: "2485dde8723650353c1cca714d7bbad5"
server_key_iv: "891b6f69b84633ca03b4577cc639baad"
server_key_version: "encryptionkey@0"
I20260927 21:27:28.071215 7019 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 21:27:28.074761 7034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:28.076238 7019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:27:28.076457 7019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-1/wal
uuid: "9f5af1173bb24003b4cac6a9beac8b91"
format_stamp: "Formatted at 2026-09-27 21:27:28 on dist-test-slave-bxm0"
server_key: "2485dde8723650353c1cca714d7bbad5"
server_key_iv: "891b6f69b84633ca03b4577cc639baad"
server_key_version: "encryptionkey@0"
I20260927 21:27:28.076658 7019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:28.100744 6880 heartbeater.cc:503] Master 127.0.97.62:39737 was elected leader, sending a full tablet report...
I20260927 21:27:28.113032 7019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:28.117697 7019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:28.117972 7019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:28.136824 7019 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:34259
I20260927 21:27:28.136868 7085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:34259 every 8 connection(s)
I20260927 21:27:28.138310 7019 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-1/data/info.pb
I20260927 21:27:28.140007 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 7019
I20260927 21:27:28.142506 7086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:28.153327 7086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:27:28 dist-test-slave-bxm0 krb5kdc[6512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 27 21:27:28 dist-test-slave-bxm0 krb5kdc[6512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544448, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 21:27:28.167474 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.142123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52594 (local address 127.0.97.61:34259)
0927 21:27:28.142886 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:28.142898 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:28.142939 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:27:28.145323 (+ 2384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:28.145381 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:28.146016 (+ 635us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:28.146560 (+ 544us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:28.148087 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.150509 (+ 2422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.152470 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.153211 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.153372 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.158570 (+ 5198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:28.158583 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:28.158602 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:28.158650 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:28.162314 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.163149 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.163158 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.163167 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.163385 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.163767 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.163772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.163775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.164568 (+ 793us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:28.164832 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:28.165458 (+ 626us) server_negotiation.cc:299] Negotiation successful
0927 21:27:28.165992 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":567,"thread_start_us":360,"threads_started":1}
I20260927 21:27:28.174093 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.156675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53412 (local address 127.0.97.62:39737)
0927 21:27:28.157162 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:28.157168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:28.158197 (+ 1029us) server_negotiation.cc:407] Connection header received
0927 21:27:28.158412 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:28.158417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:28.158513 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:28.158634 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:28.159829 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.161368 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.163152 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.163786 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.163929 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.168138 (+ 4209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:28.168146 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:28.168151 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:28.168195 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:28.171423 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.172151 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.172155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.172158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.172278 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.172710 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.172714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.172716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.173023 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:28.173138 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:28.173593 (+ 455us) server_negotiation.cc:299] Negotiation successful
0927 21:27:28.173841 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":239,"threads_started":1}
I20260927 21:27:28.174115 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.156672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.0.1:53412)
0927 21:27:28.158026 (+ 1354us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:28.158083 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:28.158256 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:28.158687 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:28.158692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:28.159388 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:28.159676 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.159684 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.161510 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.161514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.162955 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.162969 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.163978 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.163982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.164100 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.164877 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:28.164900 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:28.167931 (+ 3031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:28.171555 (+ 3624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:28.171563 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:28.171569 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:28.172000 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:28.172411 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:28.172415 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:28.172418 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:28.172568 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:28.173142 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:28.173148 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:28.173294 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 21:27:28.173477 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 21:27:28.173829 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1064,"mutex_wait_us":42,"thread_start_us":876,"threads_started":1}
I20260927 21:27:28.175793 7086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:27:28 dist-test-slave-bxm0 krb5kdc[6512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544448, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 21:27:28.188118 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.176622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34259 (local address 127.0.0.1:52602)
0927 21:27:28.176908 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:28.176922 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:28.177010 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:28.177243 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:28.177245 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:28.177550 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:28.177731 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.177739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.178696 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.178700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.179593 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.179605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.180317 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.180321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.180561 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.181232 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:28.181248 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:28.183420 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:28.186028 (+ 2608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:28.186032 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:28.186034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:28.186405 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:28.186716 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:28.186719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:28.186721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:28.186833 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:28.187294 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:28.187298 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:28.187431 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 21:27:28.187569 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 21:27:28.187794 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
I20260927 21:27:28.188203 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.176622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52602 (local address 127.0.97.61:34259)
0927 21:27:28.176839 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:28.176843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:28.177009 (+ 166us) server_negotiation.cc:407] Connection header received
0927 21:27:28.177080 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:28.177084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:28.177137 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:28.177211 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:28.177877 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.178566 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.179746 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.180179 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.180279 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.183581 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:28.183586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:28.183589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:28.183616 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:28.185917 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.186502 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.186506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.186508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.186627 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.186927 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.186930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.186932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.187165 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:28.187267 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:28.187638 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 21:27:28.187899 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":79}
I20260927 21:27:28.193166 7086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91: Bootstrap starting.
I20260927 21:27:28.196251 7086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:28.197266 7086 log.cc:824] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91: Log is configured to *not* fsync() on all Append() calls
I20260927 21:27:28.200006 7086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91: No bootstrap required, opened a new log
I20260927 21:27:28.207134 7086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } }
I20260927 21:27:28.207492 7086 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:28.207597 7086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f5af1173bb24003b4cac6a9beac8b91, State: Initialized, Role: FOLLOWER
I20260927 21:27:28.208233 7086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } }
I20260927 21:27:28.209133 7093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } } }
I20260927 21:27:28.209431 7093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:27:28.210300 7086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 21:27:28.219640 7104 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:27:28.219800 7104 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:27:28.222347 7019 master_runner.cc:428] Detected that this master 9f5af1173bb24003b4cac6a9beac8b91 is joining an existing cluster
I20260927 21:27:28.222461 7019 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:34259
I20260927 21:27:28.245450 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.228115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52614 (local address 127.0.97.61:34259)
0927 21:27:28.228263 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:28.228274 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:28.230708 (+ 2434us) server_negotiation.cc:407] Connection header received
0927 21:27:28.230834 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:28.230840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:28.230917 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:28.231023 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:28.232409 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.233572 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.235508 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.236195 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.236289 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.238782 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:28.238789 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:28.238793 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:28.238824 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:28.242331 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.243195 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.243200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.243204 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.243352 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.243768 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.243773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.243776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.244166 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:28.244295 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:28.244830 (+ 535us) server_negotiation.cc:299] Negotiation successful
0927 21:27:28.245083 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":33}
I20260927 21:27:28.245595 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.227487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53418 (local address 127.0.97.62:39737)
0927 21:27:28.227684 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:28.227691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:28.227821 (+ 130us) server_negotiation.cc:407] Connection header received
0927 21:27:28.228082 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:28.228090 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:28.228208 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:28.228284 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:28.230067 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.231209 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.234142 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.234916 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.235032 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.238065 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:28.238074 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:28.238082 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:28.238129 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:28.241964 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.242922 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.242931 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.242937 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.243093 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.243611 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.243617 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.243622 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.244040 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:28.244183 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:28.244843 (+ 660us) server_negotiation.cc:299] Negotiation successful
0927 21:27:28.245130 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 21:27:28.246870 7111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.228467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34259 (local address 127.0.0.1:52614)
0927 21:27:28.230594 (+ 2127us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:28.230614 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:28.230732 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:28.231171 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:28.231175 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:28.231893 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:28.232236 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.232247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.233761 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.233767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.235344 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.235358 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.236610 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.236617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.236775 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.237423 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:28.237455 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:28.238612 (+ 1157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:28.242494 (+ 3882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:28.242499 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:28.242503 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:28.243073 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:28.243492 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:28.243496 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:28.243500 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:28.243652 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:28.244394 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:28.244399 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:28.244539 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 21:27:28.246054 (+ 1515us) client_negotiation.cc:240] Negotiation successful
0927 21:27:28.246399 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2006,"thread_start_us":1823,"threads_started":1}
I20260927 21:27:28.247224 7110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.226968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.0.1:53418)
0927 21:27:28.227711 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:28.227738 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:28.227936 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:28.228601 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:28.228608 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:28.229396 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:28.229879 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.229896 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.231417 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.231424 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.233979 (+ 2555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.233992 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.235342 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.235350 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.235574 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.236298 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:28.236331 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:28.237829 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:28.242145 (+ 4316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:28.242153 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:28.242160 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:28.242772 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:28.243247 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:28.243255 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:28.243261 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:28.243453 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:28.244273 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:28.244279 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:28.244500 (+ 221us) client_negotiation.cc:769] Sending connection context
0927 21:27:28.246030 (+ 1530us) client_negotiation.cc:240] Negotiation successful
0927 21:27:28.246613 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":215,"spinlock_wait_cycles":27904,"thread_start_us":593,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 27 21:27:28 dist-test-slave-bxm0 krb5kdc[6512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544445, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 21:27:28.273340 7113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.254233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34259 (local address 127.0.0.1:52624)
0927 21:27:28.254912 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:28.254959 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:28.255200 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:28.255730 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:28.255737 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:28.256461 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:28.256707 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.256717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.257901 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.257909 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.259044 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.259057 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.260107 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.260113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.260376 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.261274 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:28.261321 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:28.264627 (+ 3306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:28.269693 (+ 5066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:28.269711 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:28.269719 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:28.270438 (+ 719us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:28.270994 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:28.271003 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:28.271009 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:28.271207 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:28.272001 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:28.272011 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:28.272239 (+ 228us) client_negotiation.cc:769] Sending connection context
0927 21:27:28.272475 (+ 236us) client_negotiation.cc:240] Negotiation successful
0927 21:27:28.272857 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":276,"threads_started":1}
I20260927 21:27:28.273506 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.253949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52624 (local address 127.0.97.61:34259)
0927 21:27:28.254149 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:28.254157 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:28.255054 (+ 897us) server_negotiation.cc:407] Connection header received
0927 21:27:28.255446 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:28.255452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:28.255562 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:28.255682 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:28.256874 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.257758 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.259215 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.259921 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.260097 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.264908 (+ 4811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:28.264923 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:28.264933 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:28.264986 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:28.269444 (+ 4458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.270624 (+ 1180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.270633 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.270640 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.270839 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.271364 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.271372 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.271379 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.271835 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:28.272015 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:28.272625 (+ 610us) server_negotiation.cc:299] Negotiation successful
0927 21:27:28.272956 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 21:27:28.276022 6557 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:34259: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 34259
}
attrs {
promote: true
}
}
dest_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2"
cas_config_opid_index: -1
I20260927 21:27:28.277180 6557 consensus_queue.cc:237] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: true } }
I20260927 21:27:28.280983 7117 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9f5af1173bb24003b4cac6a9beac8b91 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: true } } }
I20260927 21:27:28.282217 7061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260927 21:27:28.285293 7117 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:34259
I20260927 21:27:28.286085 7061 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 1 FOLLOWER]: Refusing update from remote peer a978f7733b8c466f9f9ef4d1acefb6d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 21:27:28.286367 7117 sys_catalog.cc:455] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: true } } }
I20260927 21:27:28.286624 7117 sys_catalog.cc:458] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:28.287045 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44991
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39737,127.0.97.61:34259,127.0.97.60:44991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:27:28.289057 7019 master.cc:537] Master@127.0.97.61:34259 shutting down...
I20260927 21:27:28.290619 7117 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 21:27:28.294150 6544 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 21:27:28.296782 6544 consensus_peers.cc:487] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 -> Peer 9f5af1173bb24003b4cac6a9beac8b91 (127.0.97.61:34259): Couldn't send request to peer 9f5af1173bb24003b4cac6a9beac8b91. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 21:27:28.302425 7019 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:27:28.302955 7019 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:27:28.303208 7019 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91: stopping tablet replica
I20260927 21:27:28.311030 7019 master.cc:552] Master@127.0.97.61:34259 shutdown complete.
I20260927 21:27:28.311142 7019 master_runner.cc:305] Clearing existing system tablet
I20260927 21:27:28.311703 7019 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:28.311825 7019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:28.320288 7019 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.008s sys 0.000s
I20260927 21:27:28.322314 7122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:28.322710 7019 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:28.323001 7019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 21:27:28.323132 7019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-1/wal
uuid: "9f5af1173bb24003b4cac6a9beac8b91"
format_stamp: "Formatted at 2026-09-27 21:27:28 on dist-test-slave-bxm0"
server_key: "2485dde8723650353c1cca714d7bbad5"
server_key_iv: "891b6f69b84633ca03b4577cc639baad"
server_key_version: "encryptionkey@0"
I20260927 21:27:28.328238 7019 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:27:28.335059 7019 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 21:27:28.335249 7019 log.cc:1196] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 21:27:28.335708 7019 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91: Deleting consensus metadata
I20260927 21:27:28.335989 7019 master_runner.cc:315] Copying system tablet from 127.0.97.62:39737
I20260927 21:27:28.338500 7019 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:39737
I20260927 21:27:28.355306 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.339371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53422 (local address 127.0.97.62:39737)
0927 21:27:28.339555 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:28.339561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:28.340040 (+ 479us) server_negotiation.cc:407] Connection header received
0927 21:27:28.340236 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:28.340243 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:28.340360 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:28.340481 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:28.341510 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.342692 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.344575 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.345341 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.345487 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.348037 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:28.348045 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:28.348051 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:28.348095 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:28.351757 (+ 3662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.352737 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.352743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.352747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.352909 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.353444 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.353450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.353453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.353867 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:28.354029 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:28.354601 (+ 572us) server_negotiation.cc:299] Negotiation successful
0927 21:27:28.354846 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 21:27:28.355475 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.339288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.0.1:53422)
0927 21:27:28.339898 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:28.339919 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:28.340090 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:28.340531 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:28.340536 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:28.341051 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:28.341341 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.341354 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.342880 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.342886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.344407 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.344422 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.345521 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.345526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.345738 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.346597 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:28.346624 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:28.347795 (+ 1171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:28.351938 (+ 4143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:28.351946 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:28.351953 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:28.352592 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:28.353059 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:28.353066 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:28.353072 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:28.353288 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:28.354019 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:28.354026 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:28.354260 (+ 234us) client_negotiation.cc:769] Sending connection context
0927 21:27:28.354471 (+ 211us) client_negotiation.cc:240] Negotiation successful
0927 21:27:28.354883 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":371,"threads_started":1}
I20260927 21:27:28.356511 6577 tablet_copy_service.cc:140] P a978f7733b8c466f9f9ef4d1acefb6d2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9f5af1173bb24003b4cac6a9beac8b91 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:53422)
I20260927 21:27:28.356796 6577 tablet_copy_service.cc:161] P a978f7733b8c466f9f9ef4d1acefb6d2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9f5af1173bb24003b4cac6a9beac8b91 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:53422: session id = 9f5af1173bb24003b4cac6a9beac8b91-00000000000000000000000000000000
I20260927 21:27:28.361230 6577 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 21:27:28.363677 7019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 21:27:28.371160 7019 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91: tablet copy: Starting download of 0 data blocks...
I20260927 21:27:28.371536 7019 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91: tablet copy: Starting download of 1 WAL segments...
I20260927 21:27:28.374846 7019 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 21:27:28.379781 6577 tablet_copy_service.cc:342] P a978f7733b8c466f9f9ef4d1acefb6d2: Request end of tablet copy session 9f5af1173bb24003b4cac6a9beac8b91-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:53422
I20260927 21:27:28.380031 6577 tablet_copy_service.cc:433] P a978f7733b8c466f9f9ef4d1acefb6d2: ending tablet copy session 9f5af1173bb24003b4cac6a9beac8b91-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9f5af1173bb24003b4cac6a9beac8b91
I20260927 21:27:28.392313 7019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:28.395185 7129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:28.399147 7130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:28.411257 7019 server_base.cc:1034] running on GCE node
W20260927 21:27:28.414011 7132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:28.414543 7019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:28.415307 7019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:28.416584 7019 hybrid_clock.cc:648] HybridClock initialized: now 1790544448416517 us; error 42 us; skew 500 ppm
I20260927 21:27:28.416836 7019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:28 dist-test-slave-bxm0 krb5kdc[6512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544448, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:28.419194 7019 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260927 21:27:28.420622 7019 webserver.cc:460] Webserver started at http://127.0.97.61:34393/ using document root <none> and password file <none>
I20260927 21:27:28.421013 7019 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:28.421101 7019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:28.424530 7019 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260927 21:27:28.426652 7139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:28.427182 7019 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:27:28.427394 7019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-1/wal
uuid: "9f5af1173bb24003b4cac6a9beac8b91"
format_stamp: "Formatted at 2026-09-27 21:27:28 on dist-test-slave-bxm0"
server_key: "2485dde8723650353c1cca714d7bbad5"
server_key_iv: "891b6f69b84633ca03b4577cc639baad"
server_key_version: "encryptionkey@0"
I20260927 21:27:28.427714 7019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:28.443547 7019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:28.448019 7019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:28.472777 7019 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:34259
I20260927 21:27:28.472865 7190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:34259 every 8 connection(s)
I20260927 21:27:28.474110 7019 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-1/data/info.pb
I20260927 21:27:28.479700 7191 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:27:28.483785 7191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91: Bootstrap starting.
I20260927 21:27:28.502774 7191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:27:28.503439 7191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91: Bootstrap complete.
I20260927 21:27:28.504551 7191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: true } }
I20260927 21:27:28.504842 7191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9f5af1173bb24003b4cac6a9beac8b91, State: Initialized, Role: LEARNER
I20260927 21:27:28.505115 7191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: true } }
I20260927 21:27:28.505990 7194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: true } } }
I20260927 21:27:28.506459 7194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [sys.catalog]: This master's current role is: LEARNER
I20260927 21:27:28.506606 7191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:27:28.514664 7205 catalog_manager.cc:1259] Loaded cluster ID: 700f24069d884b139152c6fa230b5dd4
I20260927 21:27:28.514773 7205 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91: loading cluster ID for follower catalog manager: success
I20260927 21:27:28.518247 7205 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91: acquiring CA information for follower catalog manager: success
I20260927 21:27:28.520740 7205 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:27:28.613790 7014 heartbeater.cc:503] Master 127.0.97.62:39737 was elected leader, sending a full tablet report...
W20260927 21:27:28.713238 7119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:28.713587 7119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:28.713753 7119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:28.721268 7119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:27:28.721393 7119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:28.721487 7119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:28.721592 7119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:27:28.721724 7119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:27:28.732638 7119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39737,127.0.97.61:34259,127.0.97.60:44991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:28.734095 7119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:28.735440 7119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:28.747985 7213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:28.747956 7211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:28.748858 7210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:28.749138 7113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.741373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34259 (local address 127.0.0.1:52640)
0927 21:27:28.741718 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:28.741741 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:28.741898 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:28.743082 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:28.743088 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:28.743117 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:28.743314 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.743323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.744613 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.744620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.746432 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.746450 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.748259 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.748265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.748530 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.748608 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 21:27:28.748684 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 21:27:28.748837 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260927 21:27:28.749761 7119 server_base.cc:1034] running on GCE node
I20260927 21:27:28.750144 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.742045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52640 (local address 127.0.97.61:34259)
0927 21:27:28.742760 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:28.742764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:28.742781 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:27:28.742839 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:28.742843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:28.742910 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:28.743043 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:28.743492 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.744465 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.746642 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.748065 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.749247 (+ 1182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.749552 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:28.749662 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 21:27:28.749849 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":404,"threads_started":1}
I20260927 21:27:28.750779 7119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:28.751230 7166 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260927 21:27:28.751852 7194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a978f7733b8c466f9f9ef4d1acefb6d2. Latest consensus state: current_term: 1 leader_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: true } } }
I20260927 21:27:28.752125 7194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [sys.catalog]: This master's current role is: LEARNER
I20260927 21:27:28.752693 7119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:28.753921 7119 hybrid_clock.cc:648] HybridClock initialized: now 1790544448753864 us; error 48 us; skew 500 ppm
I20260927 21:27:28.754279 7119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:28 dist-test-slave-bxm0 krb5kdc[6512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544448, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:28.757769 7119 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260927 21:27:28.759346 7119 webserver.cc:460] Webserver started at http://127.0.97.60:38163/ using document root <none> and password file <none>
I20260927 21:27:28.760099 7119 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:28.760201 7119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:28.760525 7119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:28.764034 7119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-2/data/instance:
uuid: "da163c81fef446f9bc4b8a31a02b413e"
format_stamp: "Formatted at 2026-09-27 21:27:28 on dist-test-slave-bxm0"
server_key: "ad3a3b24916eca5cea834cf2ca7569a8"
server_key_iv: "a55bceb6cf5131fcca2d47ef8567cd5a"
server_key_version: "encryptionkey@0"
I20260927 21:27:28.764838 7119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-2/wal/instance:
uuid: "da163c81fef446f9bc4b8a31a02b413e"
format_stamp: "Formatted at 2026-09-27 21:27:28 on dist-test-slave-bxm0"
server_key: "ad3a3b24916eca5cea834cf2ca7569a8"
server_key_iv: "a55bceb6cf5131fcca2d47ef8567cd5a"
server_key_version: "encryptionkey@0"
I20260927 21:27:28.770529 7119 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 21:27:28.774160 7221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:28.775923 7119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:27:28.776118 7119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-2/wal
uuid: "da163c81fef446f9bc4b8a31a02b413e"
format_stamp: "Formatted at 2026-09-27 21:27:28 on dist-test-slave-bxm0"
server_key: "ad3a3b24916eca5cea834cf2ca7569a8"
server_key_iv: "a55bceb6cf5131fcca2d47ef8567cd5a"
server_key_version: "encryptionkey@0"
I20260927 21:27:28.776314 7119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:28.805333 7119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:28.811002 7119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:28.811305 7119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:28.831070 7119 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44991
I20260927 21:27:28.831115 7272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44991 every 8 connection(s)
I20260927 21:27:28.832453 7119 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-2/data/info.pb
I20260927 21:27:28.834596 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 7119
I20260927 21:27:28.837357 7273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:28.846330 7273 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:27:28 dist-test-slave-bxm0 krb5kdc[6512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 27 21:27:28 dist-test-slave-bxm0 krb5kdc[6512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544448, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 21:27:28.860682 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.836725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42954 (local address 127.0.97.60:44991)
0927 21:27:28.837497 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:28.837512 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:28.837561 (+ 49us) server_negotiation.cc:407] Connection header received
0927 21:27:28.839748 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:28.839777 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:28.840276 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:28.840766 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:28.842243 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.844291 (+ 2048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.845774 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.846834 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.847040 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.852488 (+ 5448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:28.852495 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:28.852506 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:28.852555 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:28.856177 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.857022 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.857032 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.857042 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.857247 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.857657 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.857663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.857668 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.858321 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:28.858533 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:28.859153 (+ 620us) server_negotiation.cc:299] Negotiation successful
0927 21:27:28.859530 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":302,"threads_started":1}
I20260927 21:27:28.865996 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.849290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53430 (local address 127.0.97.62:39737)
0927 21:27:28.849526 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:28.849536 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:28.850270 (+ 734us) server_negotiation.cc:407] Connection header received
0927 21:27:28.850494 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:28.850498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:28.850560 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:28.850642 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:28.851872 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.852771 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.854961 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.855508 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.855617 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.859682 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:28.859687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:28.859691 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:28.859726 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:28.862809 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.863909 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.863913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.863916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.864046 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.864467 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.864470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.864473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.864735 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:28.864890 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:28.865333 (+ 443us) server_negotiation.cc:299] Negotiation successful
0927 21:27:28.865570 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 21:27:28.866045 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.849217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.0.1:53430)
0927 21:27:28.850110 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:28.850166 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:28.850348 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:28.850758 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:28.850768 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:28.851399 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:28.851708 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.851717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.852920 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.852926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.854744 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.854760 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.855714 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.855721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.855935 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.856651 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:28.856668 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:28.859475 (+ 2807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:28.862965 (+ 3490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:28.862974 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:28.862978 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:28.863737 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:28.864164 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:28.864170 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:28.864174 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:28.864314 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:28.864865 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:28.864873 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:28.865047 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 21:27:28.865231 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 21:27:28.865586 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":434,"threads_started":1}
I20260927 21:27:28.867735 7273 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:27:28 dist-test-slave-bxm0 krb5kdc[6512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544448, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 21:27:28.880734 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.868577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34259 (local address 127.0.0.1:52642)
0927 21:27:28.868803 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:28.868819 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:28.868909 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:28.869190 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:28.869193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:28.869516 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:28.869719 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.869728 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.870678 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.870682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.871667 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.871678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.872435 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.872447 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.872595 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.873240 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:28.873257 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:28.875314 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:28.878436 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:28.878440 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:28.878442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:28.878804 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:28.879141 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:28.879144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:28.879146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:28.879284 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:28.879924 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:28.879927 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:28.880043 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 21:27:28.880151 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 21:27:28.880430 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 21:27:28.881045 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.868690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52642 (local address 127.0.97.61:34259)
0927 21:27:28.868839 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:28.868844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:28.868861 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:27:28.869014 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:28.869018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:28.869076 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:28.869156 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:28.869840 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.870528 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.871833 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.872274 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.872388 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.875547 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:28.875553 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:28.875556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:28.875592 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:28.878304 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.878910 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.878914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.878916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.879039 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.879406 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.879411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.879413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.879704 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:28.879945 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:28.880295 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 21:27:28.880592 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 21:27:28.882170 7273 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44991 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:27:28 dist-test-slave-bxm0 krb5kdc[6512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544448, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260927 21:27:28.895078 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.882891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44991 (local address 127.0.0.1:42968)
0927 21:27:28.883074 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:28.883104 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:28.883206 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:28.883460 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:28.883463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:28.883769 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:28.883905 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.883912 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.884853 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.884857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.885800 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.885811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.886530 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.886536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.886708 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.887315 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:28.887332 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:28.889546 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:28.892961 (+ 3415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:28.892965 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:28.892967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:28.893341 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:28.893732 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:28.893736 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:28.893738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:28.893889 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:28.894359 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:28.894362 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:28.894481 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 21:27:28.894594 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 21:27:28.894787 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 21:27:28.895268 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.882917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42968 (local address 127.0.97.60:44991)
0927 21:27:28.883086 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:28.883091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:28.883178 (+ 87us) server_negotiation.cc:407] Connection header received
0927 21:27:28.883290 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:28.883294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:28.883350 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:28.883429 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:28.884018 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.884721 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.885943 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.886349 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.886447 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.889779 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:28.889783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:28.889786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:28.889815 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:28.892813 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.893448 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.893452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.893454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.893569 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.893984 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.893987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.893989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.894246 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:28.894349 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:28.894670 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 21:27:28.894934 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":59}
I20260927 21:27:28.900013 7273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e: Bootstrap starting.
I20260927 21:27:28.903380 7273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:28.904461 7273 log.cc:824] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e: Log is configured to *not* fsync() on all Append() calls
I20260927 21:27:28.907375 7273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e: No bootstrap required, opened a new log
I20260927 21:27:28.915386 7273 raft_consensus.cc:348] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44991 } }
I20260927 21:27:28.915791 7273 raft_consensus.cc:374] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:28.915931 7273 raft_consensus.cc:729] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da163c81fef446f9bc4b8a31a02b413e, State: Initialized, Role: FOLLOWER
I20260927 21:27:28.916549 7273 consensus_queue.cc:260] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44991 } }
I20260927 21:27:28.917446 7279 sys_catalog.cc:455] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44991 } } }
I20260927 21:27:28.917821 7279 sys_catalog.cc:458] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:27:28.918586 7273 sys_catalog.cc:564] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [sys.catalog]: configured and running, proceeding with master startup.
W20260927 21:27:28.928671 7290 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:27:28.928804 7290 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:27:28.931982 7119 master_runner.cc:428] Detected that this master da163c81fef446f9bc4b8a31a02b413e is joining an existing cluster
I20260927 21:27:28.932125 7119 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:44991
I20260927 21:27:28.955837 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.937420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52654 (local address 127.0.97.61:34259)
0927 21:27:28.937776 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:28.937782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:28.939275 (+ 1493us) server_negotiation.cc:407] Connection header received
0927 21:27:28.939555 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:28.939563 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:28.939644 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:28.939715 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:28.941552 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.942628 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.944599 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.945226 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.945322 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.948116 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:28.948123 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:28.948127 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:28.948165 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:28.952440 (+ 4275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.953275 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.953283 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.953289 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.953461 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.953973 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.953980 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.953983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.954367 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:28.954486 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:28.955177 (+ 691us) server_negotiation.cc:299] Negotiation successful
0927 21:27:28.955454 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218}
I20260927 21:27:28.955864 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.937824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53444 (local address 127.0.97.62:39737)
0927 21:27:28.938002 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:28.938005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:28.938019 (+ 14us) server_negotiation.cc:407] Connection header received
0927 21:27:28.938064 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:28.938067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:28.938122 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:28.938231 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:28.939621 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.941294 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.944325 (+ 3031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.944968 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.945072 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.948131 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:28.948137 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:28.948142 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:28.948178 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:28.952397 (+ 4219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.953496 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.953504 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.953510 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.953716 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.954207 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.954214 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.954217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.954563 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:28.954679 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:28.955274 (+ 595us) server_negotiation.cc:299] Negotiation successful
0927 21:27:28.955494 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 21:27:28.957018 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.936833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.0.1:53444)
0927 21:27:28.937581 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:28.937605 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:28.937764 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:28.938256 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:28.938260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:28.939042 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:28.939461 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.939474 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.941549 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.941553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.944154 (+ 2601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.944167 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.945351 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.945358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.945587 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.946532 (+ 945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:28.946548 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:28.947950 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:28.952877 (+ 4927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:28.952886 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:28.952891 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:28.953361 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:28.953875 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:28.953882 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:28.953891 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:28.954074 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:28.954758 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:28.954767 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:28.954964 (+ 197us) client_negotiation.cc:769] Sending connection context
0927 21:27:28.956140 (+ 1176us) client_negotiation.cc:240] Negotiation successful
0927 21:27:28.956457 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":23,"spinlock_wait_cycles":25856,"thread_start_us":1527,"threads_started":1}
I20260927 21:27:28.957491 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.936985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34259 (local address 127.0.0.1:52654)
0927 21:27:28.939180 (+ 2195us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:28.939199 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:28.939388 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:28.940320 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:28.940325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:28.940995 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:28.941389 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.941402 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.942847 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.942852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.944464 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.944475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.945603 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.945636 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.945850 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.946498 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:28.946524 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:28.947913 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:28.952607 (+ 4694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:28.952611 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:28.952614 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:28.953132 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:28.953636 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:28.953641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:28.953644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:28.953834 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:28.954585 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:28.954592 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:28.954837 (+ 245us) client_negotiation.cc:769] Sending connection context
0927 21:27:28.956631 (+ 1794us) client_negotiation.cc:240] Negotiation successful
0927 21:27:28.956951 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2046,"mutex_wait_us":21,"thread_start_us":1813,"threads_started":1}
I20260927 21:27:28.967373 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 6618
I20260927 21:27:28.971081 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.939618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44991 (local address 127.0.0.1:42982)
0927 21:27:28.954142 (+ 14524us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:28.954162 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:28.954322 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:28.954764 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:28.954772 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:28.955488 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:28.955843 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.955856 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.957321 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.957327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.959150 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.959168 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.960611 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.960618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.960786 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.961340 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:28.961387 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:28.962992 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:28.967716 (+ 4724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:28.967721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:28.967725 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:28.968228 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:28.968637 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:28.968644 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:28.968648 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:28.968789 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:28.969461 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:28.969466 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:28.969659 (+ 193us) client_negotiation.cc:769] Sending connection context
0927 21:27:28.969816 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 21:27:28.970115 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14369,"thread_start_us":14217,"threads_started":1}
I20260927 21:27:28.972118 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.939301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42982 (local address 127.0.97.60:44991)
0927 21:27:28.939506 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:28.939512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:28.954261 (+ 14749us) server_negotiation.cc:407] Connection header received
0927 21:27:28.954446 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:28.954451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:28.954522 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:28.954609 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:28.955989 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.957121 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.959340 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.960025 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.960150 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.963193 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:28.963200 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:28.963205 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:28.963243 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:28.967583 (+ 4340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.968371 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.968375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.968379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.968525 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.968908 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.968915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.968921 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.969333 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:28.971345 (+ 2012us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:28.971489 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 21:27:28.971711 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 21:27:28 dist-test-slave-bxm0 krb5kdc[6512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544445, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260927 21:27:28.981734 7299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.963640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42998 (local address 127.0.97.60:44991)
0927 21:27:28.964285 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:28.964292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:28.964313 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:27:28.964402 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:28.964409 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:28.964504 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:28.964618 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:28.965399 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.966907 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.970047 (+ 3140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.971188 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.973852 (+ 2664us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.975083 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:28.975087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:28.975090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:28.975121 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:28.978491 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.979231 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.979236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.979240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.979395 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.979850 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.979855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.979859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.980306 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:28.980460 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:28.980980 (+ 520us) server_negotiation.cc:299] Negotiation successful
0927 21:27:28.981213 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":603,"threads_started":1}
I20260927 21:27:28.982216 7113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.963594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44991 (local address 127.0.0.1:42998)
0927 21:27:28.963777 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:28.963799 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:28.963921 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:28.964647 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:28.964654 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:28.965082 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:28.965233 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.965241 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.967081 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.967088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.968573 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:28.968591 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.971375 (+ 2784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.971382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:28.971570 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.972245 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:28.972273 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:28.974830 (+ 2557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:28.978636 (+ 3806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:28.978641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:28.978646 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:28.979074 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:28.979541 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:28.979547 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:28.979551 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:28.979713 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:28.980460 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:28.980466 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:28.980640 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 21:27:28.980829 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 21:27:28.981299 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 21:27:28.983843 6557 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:44991: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 44991
}
attrs {
promote: true
}
}
dest_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2"
cas_config_opid_index: 5
Sep 27 21:27:28 dist-test-slave-bxm0 krb5kdc[6512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260927 21:27:28.984714 6557 consensus_queue.cc:237] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: true } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: true } }
I20260927 21:27:28.986470 7117 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER da163c81fef446f9bc4b8a31a02b413e (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: true } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: true } } }
I20260927 21:27:28.987419 7248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:28.991715 7248 raft_consensus.cc:1261] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [term 1 FOLLOWER]: Refusing update from remote peer a978f7733b8c466f9f9ef4d1acefb6d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 21:27:28.994158 7301 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:27:28.994609 7117 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:44991
I20260927 21:27:28.995185 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:28.969417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42272 (local address 127.0.97.1:33945)
0927 21:27:28.970094 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:28.970103 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:28.970135 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:27:28.970211 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:28.970223 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:28.970531 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:28.972720 (+ 2189us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:28.974246 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.975828 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.977995 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:28.978928 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:28.979073 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:28.985304 (+ 6231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:28.985321 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:28.985339 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:28.985390 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:28.989833 (+ 4443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.990779 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.990788 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.990791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.990961 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:28.991501 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:28.991509 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:28.991515 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:28.992316 (+ 801us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:28.992646 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:28.994137 (+ 1491us) server_negotiation.cc:299] Negotiation successful
0927 21:27:28.994625 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":308,"threads_started":1}
I20260927 21:27:28.995110 7302 sys_catalog.cc:455] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: true } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: true } } }
I20260927 21:27:28.995688 7302 sys_catalog.cc:458] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:28.996912 6741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:29.006227 7166 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 1 LEARNER]: Refusing update from remote peer a978f7733b8c466f9f9ef4d1acefb6d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 21:27:29.007129 7119 master.cc:537] Master@127.0.97.60:44991 shutting down...
I20260927 21:27:29.012415 7301 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260927 21:27:29.018786 6544 consensus_peers.cc:487] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 -> Peer da163c81fef446f9bc4b8a31a02b413e (127.0.97.60:44991): Couldn't send request to peer da163c81fef446f9bc4b8a31a02b413e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 21:27:29.020357 7166 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER da163c81fef446f9bc4b8a31a02b413e (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: true } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: true } } }
I20260927 21:27:29.024048 7194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: true } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: true } } }
I20260927 21:27:29.024344 7194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [sys.catalog]: This master's current role is: LEARNER
I20260927 21:27:29.024820 7302 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2: attempting to promote NON_VOTER 9f5af1173bb24003b4cac6a9beac8b91 to VOTER
I20260927 21:27:29.025813 7302 consensus_queue.cc:237] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: true } }
I20260927 21:27:29.027889 7166 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 1 LEARNER]: Refusing update from remote peer a978f7733b8c466f9f9ef4d1acefb6d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 21:27:29.028484 7303 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260927 21:27:29.028939 6544 consensus_peers.cc:487] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 -> Peer da163c81fef446f9bc4b8a31a02b413e (127.0.97.60:44991): Couldn't send request to peer da163c81fef446f9bc4b8a31a02b413e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 21:27:29.031091 7301 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9f5af1173bb24003b4cac6a9beac8b91 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: true } } }
I20260927 21:27:29.031864 7166 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9f5af1173bb24003b4cac6a9beac8b91 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: true } } }
I20260927 21:27:29.035346 7302 sys_catalog.cc:455] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: true } } }
I20260927 21:27:29.035646 7302 sys_catalog.cc:458] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:29.035377 7117 sys_catalog.cc:455] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: true } } }
I20260927 21:27:29.035990 7117 sys_catalog.cc:458] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:29.037380 7194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: true } } }
I20260927 21:27:29.037657 7194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:27:29.045043 7119 raft_consensus.cc:2229] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:27:29.045531 7119 raft_consensus.cc:2258] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:27:29.045827 7119 tablet_replica.cc:330] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e: stopping tablet replica
I20260927 21:27:29.052400 7119 master.cc:552] Master@127.0.97.60:44991 shutdown complete.
I20260927 21:27:29.052565 7119 master_runner.cc:305] Clearing existing system tablet
I20260927 21:27:29.053023 7119 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:29.053186 7119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:29.056263 7119 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260927 21:27:29.057971 7309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:29.058308 7119 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:29.058719 7119 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:27:29.058941 7119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-2/wal
uuid: "da163c81fef446f9bc4b8a31a02b413e"
format_stamp: "Formatted at 2026-09-27 21:27:28 on dist-test-slave-bxm0"
server_key: "ad3a3b24916eca5cea834cf2ca7569a8"
server_key_iv: "a55bceb6cf5131fcca2d47ef8567cd5a"
server_key_version: "encryptionkey@0"
I20260927 21:27:29.064174 7119 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:27:29.071684 7119 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 21:27:29.071997 7119 log.cc:1196] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 21:27:29.072563 7119 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e: Deleting consensus metadata
I20260927 21:27:29.072877 7119 master_runner.cc:315] Copying system tablet from 127.0.97.62:39737
I20260927 21:27:29.075560 7119 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:39737
I20260927 21:27:29.101287 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:29.076993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53458 (local address 127.0.97.62:39737)
0927 21:27:29.077209 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:29.077215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:29.086389 (+ 9174us) server_negotiation.cc:407] Connection header received
0927 21:27:29.086577 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:29.086584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:29.086675 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:29.086799 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:29.088128 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:29.089309 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:29.091287 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:29.092016 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:29.092158 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:29.094598 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:29.094605 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:29.094611 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:29.094652 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:29.098297 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:29.099145 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:29.099151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:29.099154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:29.099306 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:29.099739 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:29.099745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:29.099749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:29.100098 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:29.100241 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:29.100709 (+ 468us) server_negotiation.cc:299] Negotiation successful
0927 21:27:29.100948 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 21:27:29.101344 7315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:29.076472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.0.1:53458)
0927 21:27:29.086198 (+ 9726us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:29.086229 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:29.086430 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:29.086966 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:29.086971 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:29.087588 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:29.087912 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:29.087927 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:29.089516 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:29.089521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:29.091101 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:29.091117 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:29.092211 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:29.092217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:29.092400 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:29.093191 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:29.093217 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:29.094364 (+ 1147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:29.098457 (+ 4093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:29.098463 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:29.098467 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:29.098985 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:29.099423 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:29.099429 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:29.099433 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:29.099602 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:29.100222 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:29.100227 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:29.100423 (+ 196us) client_negotiation.cc:769] Sending connection context
0927 21:27:29.100572 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 21:27:29.100894 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9558,"thread_start_us":9399,"threads_started":1}
I20260927 21:27:29.102674 6577 tablet_copy_service.cc:140] P a978f7733b8c466f9f9ef4d1acefb6d2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer da163c81fef446f9bc4b8a31a02b413e ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:53458)
I20260927 21:27:29.102849 6577 tablet_copy_service.cc:161] P a978f7733b8c466f9f9ef4d1acefb6d2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer da163c81fef446f9bc4b8a31a02b413e at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:53458: session id = da163c81fef446f9bc4b8a31a02b413e-00000000000000000000000000000000
I20260927 21:27:29.106611 6577 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 21:27:29.109094 7119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 21:27:29.116230 7119 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e: tablet copy: Starting download of 0 data blocks...
I20260927 21:27:29.116566 7119 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e: tablet copy: Starting download of 1 WAL segments...
I20260927 21:27:29.119288 7119 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 21:27:29.120661 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 6618
I20260927 21:27:29.123934 6577 tablet_copy_service.cc:342] P a978f7733b8c466f9f9ef4d1acefb6d2: Request end of tablet copy session da163c81fef446f9bc4b8a31a02b413e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:53458
I20260927 21:27:29.124095 6577 tablet_copy_service.cc:433] P a978f7733b8c466f9f9ef4d1acefb6d2: ending tablet copy session da163c81fef446f9bc4b8a31a02b413e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer da163c81fef446f9bc4b8a31a02b413e
I20260927 21:27:29.136471 7119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:29.141877 7316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:29.142324 7317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:29.145679 7319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:29.145960 7119 server_base.cc:1034] running on GCE node
I20260927 21:27:29.145944 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33945
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39737
--webserver_port=34415
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:39737,127.0.97.61:34259
--tserver_master_addrs=127.0.97.62:39737,127.0.97.61:34259,127.0.97.60:44991 with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:27:29.146425 7119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:29.147106 7119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:29.148319 7119 hybrid_clock.cc:648] HybridClock initialized: now 1790544449148278 us; error 33 us; skew 500 ppm
I20260927 21:27:29.148587 7119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:29 dist-test-slave-bxm0 krb5kdc[6512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544449, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:29.150887 7119 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260927 21:27:29.152853 7119 webserver.cc:460] Webserver started at http://127.0.97.60:39249/ using document root <none> and password file <none>
I20260927 21:27:29.153182 7119 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:29.153277 7119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:29.156431 7119 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260927 21:27:29.158557 7327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:29.159025 7119 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260927 21:27:29.159188 7119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-2/wal
uuid: "da163c81fef446f9bc4b8a31a02b413e"
format_stamp: "Formatted at 2026-09-27 21:27:28 on dist-test-slave-bxm0"
server_key: "ad3a3b24916eca5cea834cf2ca7569a8"
server_key_iv: "a55bceb6cf5131fcca2d47ef8567cd5a"
server_key_version: "encryptionkey@0"
I20260927 21:27:29.159390 7119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:29.178987 7119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:29.183130 7119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:29.224071 7119 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44991
I20260927 21:27:29.224241 7378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44991 every 8 connection(s)
I20260927 21:27:29.225143 7119 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/master-2/data/info.pb
I20260927 21:27:29.230322 7379 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:27:29.234246 7379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e: Bootstrap starting.
I20260927 21:27:29.253279 7379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:27:29.253818 7379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e: Bootstrap complete.
I20260927 21:27:29.254937 7379 raft_consensus.cc:348] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: true } }
I20260927 21:27:29.255232 7379 raft_consensus.cc:729] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: da163c81fef446f9bc4b8a31a02b413e, State: Initialized, Role: LEARNER
I20260927 21:27:29.255479 7379 consensus_queue.cc:260] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: true } }
I20260927 21:27:29.256597 7379 sys_catalog.cc:564] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:27:29.256306 7382 sys_catalog.cc:455] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: true } } }
I20260927 21:27:29.256856 7382 sys_catalog.cc:458] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [sys.catalog]: This master's current role is: LEARNER
I20260927 21:27:29.268828 7393 catalog_manager.cc:1259] Loaded cluster ID: 700f24069d884b139152c6fa230b5dd4
I20260927 21:27:29.268962 7393 catalog_manager.cc:1552] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e: loading cluster ID for follower catalog manager: success
I20260927 21:27:29.272605 7393 catalog_manager.cc:1574] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e: acquiring CA information for follower catalog manager: success
I20260927 21:27:29.275274 7393 catalog_manager.cc:1602] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:27:29.505409 7394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:29.496650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44991 (local address 127.0.0.1:43008)
0927 21:27:29.498898 (+ 2248us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:29.498927 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:29.499072 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:29.499484 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:29.499487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:29.499507 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:29.499689 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:29.499697 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:29.501591 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:29.501595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:29.502947 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:29.502964 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:29.504794 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:29.504799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:29.504939 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:29.504968 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 21:27:29.505043 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 21:27:29.505149 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2134,"thread_start_us":2019,"threads_started":1}
I20260927 21:27:29.505553 7395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:29.496923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43008 (local address 127.0.97.60:44991)
0927 21:27:29.498750 (+ 1827us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:29.498758 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:29.499040 (+ 282us) server_negotiation.cc:407] Connection header received
0927 21:27:29.499204 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:29.499210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:29.499303 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:29.499453 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:29.499838 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:29.501404 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:29.503191 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:29.504596 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:29.504744 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:29.505029 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:29.505087 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 21:27:29.505222 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1662,"thread_start_us":1498,"threads_started":1}
I20260927 21:27:29.506685 7354 raft_consensus.cc:1203] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 21:27:29.507288 7382 sys_catalog.cc:455] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a978f7733b8c466f9f9ef4d1acefb6d2. Latest consensus state: current_term: 1 leader_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: true } } }
I20260927 21:27:29.507637 7382 sys_catalog.cc:458] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [sys.catalog]: This master's current role is: LEARNER
W20260927 21:27:29.530812 7323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:29.531138 7323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:29.531316 7323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:29.539551 7323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:29.539667 7323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:29.539836 7323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:27:29.549970 7323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34415
--tserver_master_addrs=127.0.97.62:39737,127.0.97.61:34259,127.0.97.60:44991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:29.551344 7323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:29.552724 7323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:29.567114 7400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:29.567201 7403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:29.567176 7401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:29.568365 7323 server_base.cc:1034] running on GCE node
I20260927 21:27:29.569005 7323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:29.570113 7323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:29.571350 7323 hybrid_clock.cc:648] HybridClock initialized: now 1790544449571262 us; error 71 us; skew 500 ppm
I20260927 21:27:29.571735 7323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:29 dist-test-slave-bxm0 krb5kdc[6512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544449, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:29.575353 7323 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 21:27:29.577350 7323 webserver.cc:460] Webserver started at http://127.0.97.1:34415/ using document root <none> and password file <none>
I20260927 21:27:29.578150 7323 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:29.578303 7323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:29.584417 7323 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:27:29.587713 7410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:29.589583 7323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:27:29.589811 7323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "7f60711999224fc782e173cc4e4af402"
format_stamp: "Formatted at 2026-09-27 21:27:26 on dist-test-slave-bxm0"
server_key: "3b47232ca91e68c3052a7f98f34b3a30"
server_key_iv: "73d435b7075606164cd8a2c68abec9ea"
server_key_version: "encryptionkey@0"
I20260927 21:27:29.590852 7323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:29.607939 7323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:29.612957 7323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:29.613251 7323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:29.614338 7323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:27:29.615694 7323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:29.615823 7323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:29.615965 7323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:29.616053 7323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:29.656518 7323 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33945
I20260927 21:27:29.656572 7522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33945 every 8 connection(s)
I20260927 21:27:29.658233 7323 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:27:29.662252 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 7323
I20260927 21:27:29.662609 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 6752
Sep 27 21:27:29 dist-test-slave-bxm0 krb5kdc[6512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544449, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 27 21:27:29 dist-test-slave-bxm0 krb5kdc[6512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
Sep 27 21:27:29 dist-test-slave-bxm0 krb5kdc[6512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544449, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 27 21:27:29 dist-test-slave-bxm0 krb5kdc[6512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544449, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 21:27:29.697010 7395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:29.661861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36535 (local address 127.0.97.60:44991)
0927 21:27:29.662084 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:29.662089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:29.667927 (+ 5838us) server_negotiation.cc:407] Connection header received
0927 21:27:29.668776 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:29.668787 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:29.668903 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:29.668996 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:29.673457 (+ 4461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:29.674958 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:29.679846 (+ 4888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:29.680808 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:29.680925 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:29.689724 (+ 8799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:29.689732 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:29.689739 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:29.689788 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:29.693225 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:29.694401 (+ 1176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:29.694409 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:29.694413 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:29.694578 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:29.695082 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:29.695090 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:29.695094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:29.695454 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:29.695590 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:29.696417 (+ 827us) server_negotiation.cc:299] Negotiation successful
0927 21:27:29.696648 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 21:27:29.698801 7533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:29.669450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34259 (local address 127.0.97.1:58631)
0927 21:27:29.679448 (+ 9998us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:29.679474 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:29.679639 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:29.680609 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:29.680615 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:29.681304 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:29.681724 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:29.681737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:29.683184 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:29.683189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:29.684702 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:29.684716 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:29.685871 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:29.685877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:29.686069 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:29.686694 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:29.686715 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:29.689699 (+ 2984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:29.693939 (+ 4240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:29.693944 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:29.693948 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:29.694454 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:29.694873 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:29.694877 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:29.694880 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:29.695047 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:29.695624 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:29.695634 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:29.696034 (+ 400us) client_negotiation.cc:769] Sending connection context
0927 21:27:29.696279 (+ 245us) client_negotiation.cc:240] Negotiation successful
0927 21:27:29.696869 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9823,"mutex_wait_us":4,"spinlock_wait_cycles":19328,"thread_start_us":9628,"threads_started":1}
I20260927 21:27:29.698830 7529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:29.663245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44991 (local address 127.0.97.1:36535)
0927 21:27:29.667800 (+ 4555us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:29.667825 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:29.668058 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:29.669668 (+ 1610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:29.669688 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:29.670956 (+ 1268us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:29.672107 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:29.672140 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:29.675273 (+ 3133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:29.675282 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:29.678269 (+ 2987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:29.678293 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:29.681293 (+ 3000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:29.681300 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:29.682566 (+ 1266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:29.685024 (+ 2458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:29.685057 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:29.689396 (+ 4339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:29.693437 (+ 4041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:29.693448 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:29.693471 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:29.694257 (+ 786us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:29.694714 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:29.694718 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:29.694721 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:29.694954 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:29.695668 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:29.695672 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:29.695996 (+ 324us) client_negotiation.cc:769] Sending connection context
0927 21:27:29.696277 (+ 281us) client_negotiation.cc:240] Negotiation successful
0927 21:27:29.698263 (+ 1986us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4369,"mutex_wait_us":1453,"thread_start_us":4150,"threads_started":1}
I20260927 21:27:29.699162 7532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:29.668176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58631 (local address 127.0.97.61:34259)
0927 21:27:29.678889 (+ 10713us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:29.678899 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:29.679596 (+ 697us) server_negotiation.cc:407] Connection header received
0927 21:27:29.679773 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:29.679779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:29.679873 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:29.679981 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:29.681904 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:29.683037 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:29.684884 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:29.685656 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:29.685794 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:29.689951 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:29.689959 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:29.689964 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:29.690004 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:29.693783 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:29.694607 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:29.694615 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:29.694622 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:29.694756 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:29.695182 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:29.695186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:29.695191 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:29.695485 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:29.695614 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:29.697022 (+ 1408us) server_negotiation.cc:299] Negotiation successful
0927 21:27:29.697235 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10538,"thread_start_us":10374,"threads_started":1}
I20260927 21:27:29.700184 7531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:29.663981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45684 (local address 127.0.97.2:38203)
0927 21:27:29.668602 (+ 4621us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:29.668615 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:29.673182 (+ 4567us) server_negotiation.cc:407] Connection header received
0927 21:27:29.673379 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:29.673392 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:29.673758 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:29.673981 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:29.676730 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:29.678677 (+ 1947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:29.681185 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:29.682272 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:29.682401 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:29.688980 (+ 6579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:29.688999 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:29.689018 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:29.689061 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:29.694498 (+ 5437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:29.696253 (+ 1755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:29.696264 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:29.696268 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:29.696422 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:29.696885 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:29.696890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:29.696894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:29.697323 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:29.698082 (+ 759us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:29.699507 (+ 1425us) server_negotiation.cc:299] Negotiation successful
0927 21:27:29.699772 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4441,"thread_start_us":4289,"threads_started":1}
I20260927 21:27:29.700670 7525 heartbeater.cc:347] Connected to a master server at 127.0.97.61:34259
I20260927 21:27:29.700757 7527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:29.662508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.1:57775)
0927 21:27:29.663535 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:29.663599 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:29.666773 (+ 3174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:29.681435 (+ 14662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:29.681439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:29.681910 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:29.682150 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:29.682160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:29.683536 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:29.683541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:29.685024 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:29.685037 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:29.687229 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:29.687236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:29.687435 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:29.688450 (+ 1015us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:29.688473 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:29.691331 (+ 2858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:29.695142 (+ 3811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:29.695148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:29.695152 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:29.695754 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:29.698047 (+ 2293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:29.698055 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:29.698062 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:29.698236 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:29.698880 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:29.698884 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:29.699044 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 21:27:29.699165 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 21:27:29.700116 (+ 951us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":831,"threads_started":1}
I20260927 21:27:29.700788 7523 heartbeater.cc:347] Connected to a master server at 127.0.97.60:44991
I20260927 21:27:29.701033 7523 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:29.701537 6875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:29.701869 7528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:29.663124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57775 (local address 127.0.97.62:39737)
0927 21:27:29.681121 (+ 17997us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:29.681127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:29.681146 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:27:29.681200 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:29.681207 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:29.681285 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:29.681402 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:29.682301 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:29.683394 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:29.686003 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:29.687004 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:29.687161 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:29.691540 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:29.691546 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:29.691552 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:29.691595 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:29.695025 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:29.697734 (+ 2709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:29.697739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:29.697742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:29.697904 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:29.698398 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:29.698403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:29.698405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:29.698731 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:29.701091 (+ 2360us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:29.701232 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 21:27:29.701471 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17860,"thread_start_us":18394,"threads_started":1}
I20260927 21:27:29.702016 7523 heartbeater.cc:511] Master 127.0.97.60:44991 requested a full tablet report, sending...
I20260927 21:27:29.702872 7524 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39737
I20260927 21:27:29.702984 7524 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:29.701041 7525 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:29.703497 7524 heartbeater.cc:511] Master 127.0.97.62:39737 requested a full tablet report, sending...
I20260927 21:27:29.703817 7525 heartbeater.cc:511] Master 127.0.97.61:34259 requested a full tablet report, sending...
I20260927 21:27:29.704694 7344 ts_manager.cc:194] Registered new tserver with Master: 7f60711999224fc782e173cc4e4af402 (127.0.97.1:33945)
I20260927 21:27:29.706331 6557 ts_manager.cc:194] Re-registered known tserver with Master: 7f60711999224fc782e173cc4e4af402 (127.0.97.1:33945)
I20260927 21:27:29.706581 7156 ts_manager.cc:194] Registered new tserver with Master: 7f60711999224fc782e173cc4e4af402 (127.0.97.1:33945)
I20260927 21:27:29.707674 6557 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57775
I20260927 21:27:29.791589 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 6752
I20260927 21:27:29.807834 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38203
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39737
--webserver_port=36323
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:39737,127.0.97.61:34259
--tserver_master_addrs=127.0.97.62:39737,127.0.97.61:34259,127.0.97.60:44991 with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:27:29.959340 7303 sys_catalog.cc:455] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: true } } }
I20260927 21:27:29.959806 7303 sys_catalog.cc:458] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:29.961118 7537 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2: attempting to promote NON_VOTER da163c81fef446f9bc4b8a31a02b413e to VOTER
I20260927 21:27:29.961737 7537 consensus_queue.cc:237] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: false } }
I20260927 21:27:29.963680 7354 raft_consensus.cc:1261] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [term 1 LEARNER]: Refusing update from remote peer a978f7733b8c466f9f9ef4d1acefb6d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 21:27:29.963800 7166 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 1 FOLLOWER]: Refusing update from remote peer a978f7733b8c466f9f9ef4d1acefb6d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 21:27:29.964315 7536 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 21:27:29.964733 7537 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 21:27:29.967762 7303 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, da163c81fef446f9bc4b8a31a02b413e (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: false } } }
I20260927 21:27:29.968425 7166 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, da163c81fef446f9bc4b8a31a02b413e (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: false } } }
I20260927 21:27:29.971902 7536 sys_catalog.cc:455] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: false } } }
I20260927 21:27:29.971971 7303 sys_catalog.cc:455] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: false } } }
I20260927 21:27:29.972332 7303 sys_catalog.cc:458] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:29.972570 7536 sys_catalog.cc:458] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:29.973690 7540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: false } } }
I20260927 21:27:29.974146 7540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:27:29.974431 7303 sys_catalog.cc:455] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: false } } }
I20260927 21:27:29.974738 7303 sys_catalog.cc:458] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:29.975129 7354 raft_consensus.cc:2932] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, da163c81fef446f9bc4b8a31a02b413e (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: false } } }
I20260927 21:27:29.978204 7382 sys_catalog.cc:455] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: false } } }
I20260927 21:27:29.978462 7382 sys_catalog.cc:458] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [sys.catalog]: This master's current role is: FOLLOWER
W20260927 21:27:30.151501 7535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:30.151840 7535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:30.152019 7535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:30.159875 7535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:30.160038 7535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:30.160229 7535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:27:30.170488 7535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36323
--tserver_master_addrs=127.0.97.62:39737,127.0.97.61:34259,127.0.97.60:44991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:30.171871 7535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:30.173192 7535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:30.186710 7546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:30.187004 7545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:30.187057 7548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:30.188040 7535 server_base.cc:1034] running on GCE node
I20260927 21:27:30.188671 7535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:30.189697 7535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:30.190917 7535 hybrid_clock.cc:648] HybridClock initialized: now 1790544450190854 us; error 62 us; skew 500 ppm
I20260927 21:27:30.191236 7535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:30 dist-test-slave-bxm0 krb5kdc[6512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544450, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:30.194427 7535 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 21:27:30.195990 7535 webserver.cc:460] Webserver started at http://127.0.97.2:36323/ using document root <none> and password file <none>
I20260927 21:27:30.196638 7535 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:30.196749 7535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:30.202486 7535 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 21:27:30.205045 7555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:30.206552 7535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 21:27:30.206718 7535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "4ff9eaae578a43f0aa15395e349d3117"
format_stamp: "Formatted at 2026-09-27 21:27:26 on dist-test-slave-bxm0"
server_key: "4c1d4793c366155f0e57d927739c8880"
server_key_iv: "3761cc8eb958cd127fb399badf285fc9"
server_key_version: "encryptionkey@0"
I20260927 21:27:30.207754 7535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:30.224550 7535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:30.229250 7535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:30.229540 7535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:30.230660 7535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:27:30.232133 7535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:30.232255 7535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:30.232395 7535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:30.232482 7535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:30.266649 7535 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38203
I20260927 21:27:30.266709 7667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38203 every 8 connection(s)
I20260927 21:27:30.268193 7535 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:27:30.269300 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 7535
I20260927 21:27:30.269573 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 6886
Sep 27 21:27:30 dist-test-slave-bxm0 krb5kdc[6512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
Sep 27 21:27:30 dist-test-slave-bxm0 krb5kdc[6512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544450, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 27 21:27:30 dist-test-slave-bxm0 krb5kdc[6512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544450, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 21:27:30.311554 7672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:30.271002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40364 (local address 127.0.97.3:33731)
0927 21:27:30.278218 (+ 7216us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:30.278233 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:30.285201 (+ 6968us) server_negotiation.cc:407] Connection header received
0927 21:27:30.285471 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:30.285487 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:30.285854 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:30.286044 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:30.288466 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.290023 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.291955 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.292848 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.292952 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:30.300280 (+ 7328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:30.300298 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:30.300321 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:30.300377 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:30.304313 (+ 3936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:30.305428 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:30.305439 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:30.305442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:30.305628 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:30.306459 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:30.306467 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:30.306472 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:30.306950 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:30.307236 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:30.310811 (+ 3575us) server_negotiation.cc:299] Negotiation successful
0927 21:27:30.311131 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6507,"thread_start_us":6311,"threads_started":1}
I20260927 21:27:30.313118 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:30.272642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54325 (local address 127.0.97.60:44991)
0927 21:27:30.285989 (+ 13347us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:30.285995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:30.286716 (+ 721us) server_negotiation.cc:407] Connection header received
0927 21:27:30.289344 (+ 2628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:30.289350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:30.289434 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:30.289540 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:30.292048 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.293186 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.296829 (+ 3643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.297585 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.297771 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:30.305112 (+ 7341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:30.305121 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:30.305129 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:30.305180 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:30.309309 (+ 4129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:30.310357 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:30.310362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:30.310366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:30.310521 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:30.311095 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:30.311100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:30.311103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:30.311431 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:30.311586 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:30.312648 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0927 21:27:30.312850 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13216,"thread_start_us":13904,"threads_started":1}
I20260927 21:27:30.313866 7009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:30.315497 7676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:30.271961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44991 (local address 127.0.97.2:54325)
0927 21:27:30.286037 (+ 14076us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:30.286120 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:30.288868 (+ 2748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:30.290010 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:30.290022 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:30.290955 (+ 933us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:30.291806 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:30.291833 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.293376 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.293381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:30.296622 (+ 3241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:30.296641 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.298216 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.298224 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:30.298565 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:30.299591 (+ 1026us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:30.299625 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:30.304804 (+ 5179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:30.309475 (+ 4671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:30.309485 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:30.309505 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:30.310188 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:30.310675 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:30.310681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:30.310686 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:30.310958 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:30.311738 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:30.311748 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:30.312195 (+ 447us) client_negotiation.cc:769] Sending connection context
0927 21:27:30.313335 (+ 1140us) client_negotiation.cc:240] Negotiation successful
0927 21:27:30.313920 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13695,"mutex_wait_us":6,"thread_start_us":14529,"threads_started":1}
Sep 27 21:27:30 dist-test-slave-bxm0 krb5kdc[6512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544450, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 21:27:30.317593 7668 heartbeater.cc:347] Connected to a master server at 127.0.97.60:44991
I20260927 21:27:30.317960 7668 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:30.319391 7668 heartbeater.cc:511] Master 127.0.97.60:44991 requested a full tablet report, sending...
I20260927 21:27:30.321533 7344 ts_manager.cc:194] Registered new tserver with Master: 4ff9eaae578a43f0aa15395e349d3117 (127.0.97.2:38203)
I20260927 21:27:30.321918 7673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:30.271393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40937 (local address 127.0.97.62:39737)
0927 21:27:30.281999 (+ 10606us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:30.282003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:30.286557 (+ 4554us) server_negotiation.cc:407] Connection header received
0927 21:27:30.293538 (+ 6981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:30.293548 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:30.293688 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:30.293789 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:30.299769 (+ 5980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.301234 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.303768 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.304675 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.304828 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:30.311288 (+ 6460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:30.311294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:30.311299 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:30.311343 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:30.317474 (+ 6131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:30.318451 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:30.318459 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:30.318465 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:30.318663 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:30.320214 (+ 1551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:30.320220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:30.320227 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:30.320625 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:30.320774 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:30.321296 (+ 522us) server_negotiation.cc:299] Negotiation successful
0927 21:27:30.321529 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10485,"thread_start_us":10370,"threads_started":1}
I20260927 21:27:30.321955 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:30.272988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.2:40937)
0927 21:27:30.286449 (+ 13461us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:30.286470 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:30.293172 (+ 6702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:30.294181 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:30.294189 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:30.296533 (+ 2344us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:30.299300 (+ 2767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:30.299315 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.301925 (+ 2610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.301931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:30.303605 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:30.303624 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.306161 (+ 2537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.306172 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:30.307142 (+ 970us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:30.307793 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:30.307810 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:30.311059 (+ 3249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:30.317713 (+ 6654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:30.317720 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:30.317724 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:30.318311 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:30.319782 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:30.319786 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:30.319789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:30.319942 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:30.320834 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:30.320839 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:30.321020 (+ 181us) client_negotiation.cc:769] Sending connection context
0927 21:27:30.321171 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 21:27:30.321488 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13338,"mutex_wait_us":2365,"thread_start_us":12516,"threads_started":1}
I20260927 21:27:30.324059 7669 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39737
I20260927 21:27:30.324200 7669 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:30.324707 7669 heartbeater.cc:511] Master 127.0.97.62:39737 requested a full tablet report, sending...
I20260927 21:27:30.325896 6557 ts_manager.cc:194] Re-registered known tserver with Master: 4ff9eaae578a43f0aa15395e349d3117 (127.0.97.2:38203)
I20260927 21:27:30.326728 6557 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40937
I20260927 21:27:30.327315 7679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:30.287245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39607 (local address 127.0.97.61:34259)
0927 21:27:30.305685 (+ 18440us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:30.305695 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:30.306366 (+ 671us) server_negotiation.cc:407] Connection header received
0927 21:27:30.306595 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:30.306600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:30.306693 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:30.306806 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:30.309239 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.310306 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.312590 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.313292 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.313400 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:30.319866 (+ 6466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:30.319877 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:30.319884 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:30.319966 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:30.323749 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:30.324616 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:30.324626 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:30.324633 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:30.324831 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:30.325322 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:30.325330 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:30.325337 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:30.326204 (+ 867us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:30.326297 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:30.326748 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 21:27:30.326969 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18234,"thread_start_us":18466,"threads_started":1}
I20260927 21:27:30.327410 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:30.287141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34259 (local address 127.0.97.2:39607)
0927 21:27:30.306271 (+ 19130us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:30.306290 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:30.306417 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:30.306867 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:30.306873 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:30.308649 (+ 1776us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:30.309034 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:30.309048 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.310484 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.310489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:30.312401 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:30.312421 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.313507 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.313513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:30.313769 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:30.315362 (+ 1593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:30.315383 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:30.319612 (+ 4229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:30.323914 (+ 4302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:30.323920 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:30.323923 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:30.324458 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:30.324985 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:30.324990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:30.324993 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:30.325168 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:30.326387 (+ 1219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:30.326390 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:30.326495 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 21:27:30.326646 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 21:27:30.326957 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18998,"mutex_wait_us":3,"thread_start_us":18844,"threads_started":1}
I20260927 21:27:30.328374 7670 heartbeater.cc:347] Connected to a master server at 127.0.97.61:34259
I20260927 21:27:30.328534 7670 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:30.329115 7670 heartbeater.cc:511] Master 127.0.97.61:34259 requested a full tablet report, sending...
I20260927 21:27:30.330225 7156 ts_manager.cc:194] Registered new tserver with Master: 4ff9eaae578a43f0aa15395e349d3117 (127.0.97.2:38203)
I20260927 21:27:30.408418 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 6886
I20260927 21:27:30.424944 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:33731
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39737
--webserver_port=36815
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:39737,127.0.97.61:34259
--tserver_master_addrs=127.0.97.62:39737,127.0.97.61:34259,127.0.97.60:44991 with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:27:30.710721 7524 heartbeater.cc:503] Master 127.0.97.62:39737 was elected leader, sending a full tablet report...
W20260927 21:27:30.766893 7681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:30.767222 7681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:30.767381 7681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:30.774664 7681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:30.774833 7681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:30.774968 7681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:27:30.785754 7681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:33731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36815
--tserver_master_addrs=127.0.97.62:39737,127.0.97.61:34259,127.0.97.60:44991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:30.787040 7681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:30.788236 7681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:30.801321 7687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:30.801275 7686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:30.801347 7689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:30.802264 7681 server_base.cc:1034] running on GCE node
I20260927 21:27:30.802825 7681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:30.803736 7681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:30.804936 7681 hybrid_clock.cc:648] HybridClock initialized: now 1790544450804893 us; error 32 us; skew 500 ppm
I20260927 21:27:30.805313 7681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:30 dist-test-slave-bxm0 krb5kdc[6512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544450, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:30.809017 7681 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 21:27:30.810691 7681 webserver.cc:460] Webserver started at http://127.0.97.3:36815/ using document root <none> and password file <none>
I20260927 21:27:30.811398 7681 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:30.811574 7681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:30.817155 7681 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:27:30.819900 7696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:30.821309 7681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 21:27:30.821482 7681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "277d8b80cca4431c9917b2ea4570c077"
format_stamp: "Formatted at 2026-09-27 21:27:27 on dist-test-slave-bxm0"
server_key: "cca73652a7ce2d2d9e10a0037aca61f5"
server_key_iv: "73f5d0b868e78ec3658b98e853004577"
server_key_version: "encryptionkey@0"
I20260927 21:27:30.822594 7681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:30.858803 7681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:30.863516 7681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:30.863751 7681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:30.864674 7681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:27:30.866055 7681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:30.866179 7681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:30.866304 7681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:30.866398 7681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:30.900283 7681 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33731
I20260927 21:27:30.900377 7808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33731 every 8 connection(s)
I20260927 21:27:30.901723 7681 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:27:30.907327 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 7681
Sep 27 21:27:30 dist-test-slave-bxm0 krb5kdc[6512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544450, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 27 21:27:30 dist-test-slave-bxm0 krb5kdc[6512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544450, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 21:27:30.936422 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:30.904675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43037 (local address 127.0.97.60:44991)
0927 21:27:30.905318 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:30.905322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:30.906686 (+ 1364us) server_negotiation.cc:407] Connection header received
0927 21:27:30.910395 (+ 3709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:30.910404 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:30.910507 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:30.910639 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:30.913101 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.914021 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.916756 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.917401 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.917527 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:30.926910 (+ 9383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:30.926921 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:30.926929 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:30.926989 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:30.931326 (+ 4337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:30.932661 (+ 1335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:30.932670 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:30.932677 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:30.932872 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:30.934377 (+ 1505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:30.934382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:30.934385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:30.934809 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:30.934941 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:30.935850 (+ 909us) server_negotiation.cc:299] Negotiation successful
0927 21:27:30.936078 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":444,"threads_started":1}
I20260927 21:27:30.936802 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:30.907245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53803 (local address 127.0.97.61:34259)
0927 21:27:30.909295 (+ 2050us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:30.909305 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:30.909344 (+ 39us) server_negotiation.cc:407] Connection header received
0927 21:27:30.909843 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:30.909853 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:30.909990 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:30.910135 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:30.913082 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.914585 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.921700 (+ 7115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.922575 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.922727 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:30.928723 (+ 5996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:30.928744 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:30.928752 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:30.928816 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:30.933245 (+ 4429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:30.934366 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:30.934375 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:30.934381 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:30.934577 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:30.934988 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:30.934995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:30.935001 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:30.935413 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:30.935607 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:30.935979 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 21:27:30.936256 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1897,"thread_start_us":1773,"threads_started":1}
I20260927 21:27:30.938823 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:30.905539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44991 (local address 127.0.97.3:43037)
0927 21:27:30.906329 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:30.906383 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:30.908831 (+ 2448us) spinlock_profiling.cc:243] Waited 1.15 ms on lock 0x61200002c628. stack: 00007fa0657a066e 00007fa0657a0430 00007fa0648275ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007fa0657960fd 00007fa065795d1e 00007fa07869f3cd 00007fa07869f36c 00007fa07866bb3b 00007fa06c1a5536 00007fa06c1a3d9e 00007fa06c22bdca 00007fa06c229f68 00007fa06c25d82f
0927 21:27:30.909470 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:30.910670 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:30.910676 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:30.911577 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:30.912876 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:30.912903 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.914275 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.914283 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:30.916606 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:30.916628 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.918749 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.918756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:30.919154 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:30.920068 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:30.920111 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:30.924525 (+ 4414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:30.931562 (+ 7037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:30.931577 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:30.931616 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:30.932456 (+ 840us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:30.933064 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:30.933072 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:30.933078 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:30.933385 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:30.935038 (+ 1653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:30.935048 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:30.935386 (+ 338us) client_negotiation.cc:769] Sending connection context
0927 21:27:30.937225 (+ 1839us) client_negotiation.cc:240] Negotiation successful
0927 21:27:30.937499 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":174,"spinlock_wait_cycles":2691200,"thread_start_us":505,"threads_started":1}
I20260927 21:27:30.939234 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:30.906990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34259 (local address 127.0.97.3:53803)
0927 21:27:30.907613 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:30.907650 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:30.909470 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:30.910401 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:30.910413 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:30.911577 (+ 1164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:30.912876 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:30.912903 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.916791 (+ 3888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.916796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:30.918555 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:30.918575 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.922832 (+ 4257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.922842 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:30.923095 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:30.924896 (+ 1801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:30.924928 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:30.928506 (+ 3578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:30.933523 (+ 5017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:30.933530 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:30.933536 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:30.934223 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:30.934734 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:30.934740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:30.934744 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:30.934888 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:30.935580 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:30.935584 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:30.935718 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 21:27:30.935870 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 21:27:30.936398 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":106,"spinlock_wait_cycles":3968,"thread_start_us":388,"threads_started":1}
I20260927 21:27:30.940935 7809 heartbeater.cc:347] Connected to a master server at 127.0.97.60:44991
I20260927 21:27:30.941053 7811 heartbeater.cc:347] Connected to a master server at 127.0.97.61:34259
I20260927 21:27:30.941290 7811 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:30.941293 7809 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:30.942335 7811 heartbeater.cc:511] Master 127.0.97.61:34259 requested a full tablet report, sending...
I20260927 21:27:30.943346 7809 heartbeater.cc:511] Master 127.0.97.60:44991 requested a full tablet report, sending...
I20260927 21:27:30.944222 7156 ts_manager.cc:194] Registered new tserver with Master: 277d8b80cca4431c9917b2ea4570c077 (127.0.97.3:33731)
I20260927 21:27:30.944522 7344 ts_manager.cc:194] Registered new tserver with Master: 277d8b80cca4431c9917b2ea4570c077 (127.0.97.3:33731)
Sep 27 21:27:30 dist-test-slave-bxm0 krb5kdc[6512](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544450, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 21:27:30.955874 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:30.904658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60141 (local address 127.0.97.62:39737)
0927 21:27:30.905115 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:30.905121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:30.906503 (+ 1382us) server_negotiation.cc:407] Connection header received
0927 21:27:30.909976 (+ 3473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:30.909982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:30.910071 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:30.910202 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:30.914798 (+ 4596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.915893 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.937292 (+ 21399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.938026 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.938171 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:30.948400 (+ 10229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:30.948406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:30.948411 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:30.948450 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:30.952056 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:30.953701 (+ 1645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:30.953706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:30.953709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:30.953866 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:30.954323 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:30.954329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:30.954332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:30.954648 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:30.954784 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:30.955275 (+ 491us) server_negotiation.cc:299] Negotiation successful
0927 21:27:30.955517 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":231,"threads_started":1}
I20260927 21:27:30.956516 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:30.905321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.3:60141)
0927 21:27:30.906331 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:30.906389 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:30.909861 (+ 3472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:30.911206 (+ 1345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:30.911219 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:30.911801 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:30.912896 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:30.912915 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.925903 (+ 12988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.925912 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:30.937132 (+ 11220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:30.937149 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.944047 (+ 6898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.944057 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:30.944343 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:30.945141 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:30.945172 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:30.948214 (+ 3042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:30.952180 (+ 3966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:30.952194 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:30.952201 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:30.952751 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:30.954002 (+ 1251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:30.954008 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:30.954013 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:30.954217 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:30.954799 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:30.954805 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:30.954976 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 21:27:30.955128 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 21:27:30.955443 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":311,"spinlock_wait_cycles":3840,"thread_start_us":471,"threads_started":1}
I20260927 21:27:30.957734 7810 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39737
I20260927 21:27:30.957913 7810 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:30.958434 7810 heartbeater.cc:511] Master 127.0.97.62:39737 requested a full tablet report, sending...
I20260927 21:27:30.960096 6557 ts_manager.cc:194] Re-registered known tserver with Master: 277d8b80cca4431c9917b2ea4570c077 (127.0.97.3:33731)
I20260927 21:27:30.961298 6557 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60141
I20260927 21:27:30.966017 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:30.938028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43022 (local address 127.0.97.60:44991)
0927 21:27:30.938190 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:30.938196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:30.948706 (+ 10510us) server_negotiation.cc:407] Connection header received
0927 21:27:30.948990 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:30.948996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:30.949119 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:30.949193 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:30.952798 (+ 3605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.953521 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.955533 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.956203 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.956310 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:30.959536 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:30.959545 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:30.959549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:30.959588 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:30.962951 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:30.963763 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:30.963768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:30.963772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:30.963922 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:30.964329 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:30.964334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:30.964338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:30.964689 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:30.964955 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:30.965330 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 21:27:30.965778 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 21:27:30.966521 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:30.916231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53460 (local address 127.0.97.62:39737)
0927 21:27:30.934396 (+ 18165us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:30.934404 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:30.938798 (+ 4394us) server_negotiation.cc:407] Connection header received
0927 21:27:30.938865 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:30.938869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:30.938956 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:30.939066 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:30.944737 (+ 5671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.945904 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.947447 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.948110 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.948235 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:30.954062 (+ 5827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:30.954068 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:30.954073 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:30.954107 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:30.960609 (+ 6502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:30.961521 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:30.961526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:30.961531 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:30.961711 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:30.962139 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:30.962144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:30.962147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:30.962491 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:30.962713 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:30.965817 (+ 3104us) server_negotiation.cc:299] Negotiation successful
0927 21:27:30.966117 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17980,"thread_start_us":18179,"threads_started":1}
I20260927 21:27:30.969004 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:30.916724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52660 (local address 127.0.97.61:34259)
0927 21:27:30.937071 (+ 20347us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:30.937076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:30.948014 (+ 10938us) server_negotiation.cc:407] Connection header received
0927 21:27:30.948203 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:30.948209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:30.948327 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:30.948435 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:30.952787 (+ 4352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.953980 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.955750 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:30.956421 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:30.956547 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:30.962396 (+ 5849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:30.962404 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:30.962408 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:30.962452 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:30.965914 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:30.966721 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:30.966729 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:30.966732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:30.966884 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:30.967320 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:30.967325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:30.967329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:30.967684 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:30.967885 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:30.968215 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 21:27:30.968437 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20235,"thread_start_us":23115,"threads_started":1}
I20260927 21:27:30.970907 6557 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:27:30.979099 6557 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:27:31.028580 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:31.002229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33945 (local address 127.0.0.1:42278)
0927 21:27:31.018959 (+ 16730us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:31.018984 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:31.019139 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:31.020358 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:31.020363 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:31.020393 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:31.020818 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:31.020834 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.023340 (+ 2506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.023346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:31.025742 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:31.025764 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.027669 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.027676 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:31.027886 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:31.027925 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 21:27:31.028067 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 21:27:31.028196 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16562,"thread_start_us":16359,"threads_started":1}
I20260927 21:27:31.029417 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:31.002716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42278 (local address 127.0.97.1:33945)
0927 21:27:31.019741 (+ 17025us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:31.019751 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:31.019795 (+ 44us) server_negotiation.cc:407] Connection header received
0927 21:27:31.019885 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:31.019892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:31.020115 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:31.020379 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:31.021088 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.023143 (+ 2055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.026016 (+ 2873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.027404 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.027582 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:31.027839 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:31.028919 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0927 21:27:31.029136 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16849,"thread_start_us":16729,"threads_started":1}
I20260927 21:27:31.030576 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:31.001285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33731 (local address 127.0.0.1:40378)
0927 21:27:31.017999 (+ 16714us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:31.018033 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:31.018231 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:31.022286 (+ 4055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:31.022293 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:31.022329 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:31.022784 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:31.022798 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.024992 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.025000 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:31.027734 (+ 2734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:31.027745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.029741 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.029748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:31.029941 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:31.029978 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 21:27:31.030048 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 21:27:31.030195 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16498,"mutex_wait_us":150,"thread_start_us":531,"threads_started":1}
I20260927 21:27:31.031097 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:31.001067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38203 (local address 127.0.0.1:45696)
0927 21:27:31.002080 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:31.002117 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:31.002565 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:31.022029 (+ 19464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:31.022034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:31.022061 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:31.022489 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:31.022505 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.025054 (+ 2549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.025063 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:31.027729 (+ 2666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:31.027745 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.030046 (+ 2301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.030054 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:31.030251 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:31.030292 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 21:27:31.030401 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 21:27:31.030569 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":177,"thread_start_us":16518,"threads_started":1}
I20260927 21:27:31.031339 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:31.001655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45696 (local address 127.0.97.2:38203)
0927 21:27:31.021462 (+ 19807us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:31.021468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:31.021492 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:27:31.021562 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:31.021567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:31.021820 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:31.022039 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:31.022726 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.024909 (+ 2183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.027925 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.029739 (+ 1814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.029882 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:31.030101 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:31.030471 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 21:27:31.030671 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19660,"thread_start_us":19538,"threads_started":1}
I20260927 21:27:31.032598 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:31.001788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40378 (local address 127.0.97.3:33731)
0927 21:27:31.021144 (+ 19356us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:31.021155 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:31.021196 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:27:31.021317 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:31.021327 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:31.022036 (+ 709us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:31.022897 (+ 861us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:31.023223 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.024798 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.027929 (+ 3131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.029500 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.031346 (+ 1846us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:31.031587 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:31.031642 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 21:27:31.031826 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19158,"thread_start_us":18977,"threads_started":1}
I20260927 21:27:31.034978 7458 tablet_service.cc:1467] Processing CreateTablet for tablet d734ddb17ce045a686625d0396f8a06a (DEFAULT_TABLE table=test-table [id=e9236fb62cb441c7a57c7292782983d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:27:31.035532 7744 tablet_service.cc:1467] Processing CreateTablet for tablet d734ddb17ce045a686625d0396f8a06a (DEFAULT_TABLE table=test-table [id=e9236fb62cb441c7a57c7292782983d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:27:31.035948 7603 tablet_service.cc:1467] Processing CreateTablet for tablet d734ddb17ce045a686625d0396f8a06a (DEFAULT_TABLE table=test-table [id=e9236fb62cb441c7a57c7292782983d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:27:31.036644 7458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d734ddb17ce045a686625d0396f8a06a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:31.036827 7744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d734ddb17ce045a686625d0396f8a06a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:31.037303 7603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d734ddb17ce045a686625d0396f8a06a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:31.049417 7845 tablet_bootstrap.cc:492] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077: Bootstrap starting.
I20260927 21:27:31.053807 7846 tablet_bootstrap.cc:492] T d734ddb17ce045a686625d0396f8a06a P 4ff9eaae578a43f0aa15395e349d3117: Bootstrap starting.
I20260927 21:27:31.054349 7845 tablet_bootstrap.cc:654] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:31.055757 7845 log.cc:824] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077: Log is configured to *not* fsync() on all Append() calls
I20260927 21:27:31.057610 7846 tablet_bootstrap.cc:654] T d734ddb17ce045a686625d0396f8a06a P 4ff9eaae578a43f0aa15395e349d3117: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:31.059043 7846 log.cc:824] T d734ddb17ce045a686625d0396f8a06a P 4ff9eaae578a43f0aa15395e349d3117: Log is configured to *not* fsync() on all Append() calls
I20260927 21:27:31.059377 7845 tablet_bootstrap.cc:492] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077: No bootstrap required, opened a new log
I20260927 21:27:31.059770 7845 ts_tablet_manager.cc:1397] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260927 21:27:31.057238 7847 tablet_bootstrap.cc:492] T d734ddb17ce045a686625d0396f8a06a P 7f60711999224fc782e173cc4e4af402: Bootstrap starting.
I20260927 21:27:31.062849 7846 tablet_bootstrap.cc:492] T d734ddb17ce045a686625d0396f8a06a P 4ff9eaae578a43f0aa15395e349d3117: No bootstrap required, opened a new log
I20260927 21:27:31.063169 7846 ts_tablet_manager.cc:1397] T d734ddb17ce045a686625d0396f8a06a P 4ff9eaae578a43f0aa15395e349d3117: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260927 21:27:31.063521 7847 tablet_bootstrap.cc:654] T d734ddb17ce045a686625d0396f8a06a P 7f60711999224fc782e173cc4e4af402: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:31.065135 7847 log.cc:824] T d734ddb17ce045a686625d0396f8a06a P 7f60711999224fc782e173cc4e4af402: Log is configured to *not* fsync() on all Append() calls
I20260927 21:27:31.072436 7845 raft_consensus.cc:348] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff9eaae578a43f0aa15395e349d3117" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38203 } } peers { permanent_uuid: "277d8b80cca4431c9917b2ea4570c077" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33731 } } peers { permanent_uuid: "7f60711999224fc782e173cc4e4af402" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33945 } }
I20260927 21:27:31.072494 7846 raft_consensus.cc:348] T d734ddb17ce045a686625d0396f8a06a P 4ff9eaae578a43f0aa15395e349d3117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff9eaae578a43f0aa15395e349d3117" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38203 } } peers { permanent_uuid: "277d8b80cca4431c9917b2ea4570c077" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33731 } } peers { permanent_uuid: "7f60711999224fc782e173cc4e4af402" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33945 } }
I20260927 21:27:31.072883 7845 raft_consensus.cc:374] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:31.072988 7845 raft_consensus.cc:729] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 277d8b80cca4431c9917b2ea4570c077, State: Initialized, Role: FOLLOWER
I20260927 21:27:31.073028 7846 raft_consensus.cc:374] T d734ddb17ce045a686625d0396f8a06a P 4ff9eaae578a43f0aa15395e349d3117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:31.073153 7846 raft_consensus.cc:729] T d734ddb17ce045a686625d0396f8a06a P 4ff9eaae578a43f0aa15395e349d3117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ff9eaae578a43f0aa15395e349d3117, State: Initialized, Role: FOLLOWER
I20260927 21:27:31.073843 7846 consensus_queue.cc:260] T d734ddb17ce045a686625d0396f8a06a P 4ff9eaae578a43f0aa15395e349d3117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff9eaae578a43f0aa15395e349d3117" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38203 } } peers { permanent_uuid: "277d8b80cca4431c9917b2ea4570c077" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33731 } } peers { permanent_uuid: "7f60711999224fc782e173cc4e4af402" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33945 } }
I20260927 21:27:31.073814 7845 consensus_queue.cc:260] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff9eaae578a43f0aa15395e349d3117" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38203 } } peers { permanent_uuid: "277d8b80cca4431c9917b2ea4570c077" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33731 } } peers { permanent_uuid: "7f60711999224fc782e173cc4e4af402" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33945 } }
I20260927 21:27:31.075392 7847 tablet_bootstrap.cc:492] T d734ddb17ce045a686625d0396f8a06a P 7f60711999224fc782e173cc4e4af402: No bootstrap required, opened a new log
I20260927 21:27:31.075726 7845 ts_tablet_manager.cc:1428] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077: Time spent starting tablet: real 0.016s user 0.008s sys 0.007s
I20260927 21:27:31.075747 7847 ts_tablet_manager.cc:1397] T d734ddb17ce045a686625d0396f8a06a P 7f60711999224fc782e173cc4e4af402: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.000s
I20260927 21:27:31.076108 7810 heartbeater.cc:503] Master 127.0.97.62:39737 was elected leader, sending a full tablet report...
I20260927 21:27:31.076332 7846 ts_tablet_manager.cc:1428] T d734ddb17ce045a686625d0396f8a06a P 4ff9eaae578a43f0aa15395e349d3117: Time spent starting tablet: real 0.013s user 0.008s sys 0.006s
I20260927 21:27:31.077485 7669 heartbeater.cc:503] Master 127.0.97.62:39737 was elected leader, sending a full tablet report...
I20260927 21:27:31.086874 7847 raft_consensus.cc:348] T d734ddb17ce045a686625d0396f8a06a P 7f60711999224fc782e173cc4e4af402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff9eaae578a43f0aa15395e349d3117" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38203 } } peers { permanent_uuid: "277d8b80cca4431c9917b2ea4570c077" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33731 } } peers { permanent_uuid: "7f60711999224fc782e173cc4e4af402" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33945 } }
I20260927 21:27:31.087407 7847 raft_consensus.cc:374] T d734ddb17ce045a686625d0396f8a06a P 7f60711999224fc782e173cc4e4af402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:31.087514 7847 raft_consensus.cc:729] T d734ddb17ce045a686625d0396f8a06a P 7f60711999224fc782e173cc4e4af402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f60711999224fc782e173cc4e4af402, State: Initialized, Role: FOLLOWER
I20260927 21:27:31.088111 7847 consensus_queue.cc:260] T d734ddb17ce045a686625d0396f8a06a P 7f60711999224fc782e173cc4e4af402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff9eaae578a43f0aa15395e349d3117" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38203 } } peers { permanent_uuid: "277d8b80cca4431c9917b2ea4570c077" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33731 } } peers { permanent_uuid: "7f60711999224fc782e173cc4e4af402" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33945 } }
I20260927 21:27:31.089928 7847 ts_tablet_manager.cc:1428] T d734ddb17ce045a686625d0396f8a06a P 7f60711999224fc782e173cc4e4af402: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260927 21:27:31.117430 7851 raft_consensus.cc:482] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:27:31.117820 7851 raft_consensus.cc:504] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff9eaae578a43f0aa15395e349d3117" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38203 } } peers { permanent_uuid: "277d8b80cca4431c9917b2ea4570c077" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33731 } } peers { permanent_uuid: "7f60711999224fc782e173cc4e4af402" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33945 } }
I20260927 21:27:31.119587 7851 leader_election.cc:288] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ff9eaae578a43f0aa15395e349d3117 (127.0.97.2:38203), 7f60711999224fc782e173cc4e4af402 (127.0.97.1:33945)
I20260927 21:27:31.125726 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:31.120067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35255 (local address 127.0.97.2:38203)
0927 21:27:31.120268 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:31.120277 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:31.120299 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:27:31.120398 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:31.120403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:31.120478 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:31.120583 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:31.121063 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.121992 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.123837 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.124696 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.124827 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:31.124922 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:31.125293 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 21:27:31.125438 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:27:31.125758 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:31.120463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37925 (local address 127.0.97.1:33945)
0927 21:27:31.120692 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:31.120698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:31.120715 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:27:31.120780 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:31.120786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:31.120872 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:31.120969 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:31.121420 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.122233 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.123951 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.124710 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.124860 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:31.124953 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:31.125365 (+ 412us) server_negotiation.cc:299] Negotiation successful
0927 21:27:31.125508 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:27:31.125739 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:31.119769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38203 (local address 127.0.97.3:35255)
0927 21:27:31.119956 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:31.119983 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:31.120144 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:31.120617 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:31.120622 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:31.120652 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:31.120849 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:31.120861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.122130 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.122136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:31.123666 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:31.123682 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.124868 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.124873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:31.125117 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:31.125160 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 21:27:31.125232 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 21:27:31.125380 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":23}
I20260927 21:27:31.125787 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:31.120022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33945 (local address 127.0.97.3:37925)
0927 21:27:31.120168 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:31.120185 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:31.120257 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:31.121040 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:31.121045 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:31.121073 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:31.121264 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:31.121276 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.122361 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.122365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:31.123782 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:31.123799 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.124902 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.124907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:31.125183 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:31.125228 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 21:27:31.125323 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 21:27:31.125453 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":38}
I20260927 21:27:31.126829 7478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d734ddb17ce045a686625d0396f8a06a" candidate_uuid: "277d8b80cca4431c9917b2ea4570c077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f60711999224fc782e173cc4e4af402" is_pre_election: true
I20260927 21:27:31.127439 7478 raft_consensus.cc:2454] T d734ddb17ce045a686625d0396f8a06a P 7f60711999224fc782e173cc4e4af402 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 277d8b80cca4431c9917b2ea4570c077 in term 0.
I20260927 21:27:31.128147 7700 leader_election.cc:302] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 277d8b80cca4431c9917b2ea4570c077, 7f60711999224fc782e173cc4e4af402; no voters:
I20260927 21:27:31.129132 7851 raft_consensus.cc:2781] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:27:31.129012 7623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d734ddb17ce045a686625d0396f8a06a" candidate_uuid: "277d8b80cca4431c9917b2ea4570c077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ff9eaae578a43f0aa15395e349d3117" is_pre_election: true
I20260927 21:27:31.129274 7851 raft_consensus.cc:482] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:27:31.129377 7851 raft_consensus.cc:3037] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:31.129639 7623 raft_consensus.cc:2454] T d734ddb17ce045a686625d0396f8a06a P 4ff9eaae578a43f0aa15395e349d3117 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 277d8b80cca4431c9917b2ea4570c077 in term 0.
I20260927 21:27:31.131717 7851 raft_consensus.cc:504] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff9eaae578a43f0aa15395e349d3117" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38203 } } peers { permanent_uuid: "277d8b80cca4431c9917b2ea4570c077" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33731 } } peers { permanent_uuid: "7f60711999224fc782e173cc4e4af402" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33945 } }
I20260927 21:27:31.132282 7851 leader_election.cc:288] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077 [CANDIDATE]: Term 1 election: Requested vote from peers 4ff9eaae578a43f0aa15395e349d3117 (127.0.97.2:38203), 7f60711999224fc782e173cc4e4af402 (127.0.97.1:33945)
I20260927 21:27:31.132781 7623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d734ddb17ce045a686625d0396f8a06a" candidate_uuid: "277d8b80cca4431c9917b2ea4570c077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ff9eaae578a43f0aa15395e349d3117"
I20260927 21:27:31.132818 7478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d734ddb17ce045a686625d0396f8a06a" candidate_uuid: "277d8b80cca4431c9917b2ea4570c077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f60711999224fc782e173cc4e4af402"
I20260927 21:27:31.132975 7623 raft_consensus.cc:3037] T d734ddb17ce045a686625d0396f8a06a P 4ff9eaae578a43f0aa15395e349d3117 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:31.133097 7478 raft_consensus.cc:3037] T d734ddb17ce045a686625d0396f8a06a P 7f60711999224fc782e173cc4e4af402 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:31.135574 7623 raft_consensus.cc:2454] T d734ddb17ce045a686625d0396f8a06a P 4ff9eaae578a43f0aa15395e349d3117 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 277d8b80cca4431c9917b2ea4570c077 in term 1.
I20260927 21:27:31.135578 7478 raft_consensus.cc:2454] T d734ddb17ce045a686625d0396f8a06a P 7f60711999224fc782e173cc4e4af402 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 277d8b80cca4431c9917b2ea4570c077 in term 1.
I20260927 21:27:31.136058 7700 leader_election.cc:302] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 277d8b80cca4431c9917b2ea4570c077, 4ff9eaae578a43f0aa15395e349d3117, 7f60711999224fc782e173cc4e4af402; no voters:
I20260927 21:27:31.136386 7851 raft_consensus.cc:2781] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:27:31.136864 7851 raft_consensus.cc:686] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077 [term 1 LEADER]: Becoming Leader. State: Replica: 277d8b80cca4431c9917b2ea4570c077, State: Running, Role: LEADER
I20260927 21:27:31.137428 7851 consensus_queue.cc:237] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff9eaae578a43f0aa15395e349d3117" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38203 } } peers { permanent_uuid: "277d8b80cca4431c9917b2ea4570c077" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33731 } } peers { permanent_uuid: "7f60711999224fc782e173cc4e4af402" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33945 } }
I20260927 21:27:31.142177 6556 catalog_manager.cc:5495] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077 reported cstate change: term changed from 0 to 1, leader changed from <none> to 277d8b80cca4431c9917b2ea4570c077 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "277d8b80cca4431c9917b2ea4570c077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff9eaae578a43f0aa15395e349d3117" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "277d8b80cca4431c9917b2ea4570c077" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f60711999224fc782e173cc4e4af402" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33945 } health_report { overall_health: UNKNOWN } } }
W20260927 21:27:31.154775 7812 tablet.cc:2369] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:27:31.162506 7526 tablet.cc:2369] T d734ddb17ce045a686625d0396f8a06a P 7f60711999224fc782e173cc4e4af402: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:27:31.171994 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:31.166837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40386 (local address 127.0.97.3:33731)
0927 21:27:31.167075 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:31.167081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:31.167102 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:27:31.167305 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:31.167309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:31.167397 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:31.167550 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:27:31.167911 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.168598 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.169930 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.170319 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.170444 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:31.170915 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:27:31.171419 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:27:31.171584 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:31.171638 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 21:27:31.171716 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 21:27:31.180661 7478 raft_consensus.cc:1261] T d734ddb17ce045a686625d0396f8a06a P 7f60711999224fc782e173cc4e4af402 [term 1 FOLLOWER]: Refusing update from remote peer 277d8b80cca4431c9917b2ea4570c077: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:27:31.180681 7623 raft_consensus.cc:1261] T d734ddb17ce045a686625d0396f8a06a P 4ff9eaae578a43f0aa15395e349d3117 [term 1 FOLLOWER]: Refusing update from remote peer 277d8b80cca4431c9917b2ea4570c077: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:27:31.181773 7854 consensus_queue.cc:1035] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f60711999224fc782e173cc4e4af402" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:27:31.182116 7851 consensus_queue.cc:1035] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ff9eaae578a43f0aa15395e349d3117" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:27:31.196024 7857 mvcc.cc:205] Tried to move back new op lower bound from 7334070072025989120 to 7334070071863353344. Current Snapshot: MvccSnapshot[applied={T|T < 7334070072025989120}]
I20260927 21:27:31.217636 6557 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:27:31.218026 6557 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:27:31.225556 6557 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a978f7733b8c466f9f9ef4d1acefb6d2: Sending DeleteTablet for 3 replicas of tablet d734ddb17ce045a686625d0396f8a06a
I20260927 21:27:31.226812 7603 tablet_service.cc:1514] Processing DeleteTablet for tablet d734ddb17ce045a686625d0396f8a06a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:27:31 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45696
I20260927 21:27:31.227813 7866 tablet_replica.cc:330] T d734ddb17ce045a686625d0396f8a06a P 4ff9eaae578a43f0aa15395e349d3117: stopping tablet replica
I20260927 21:27:31.228317 7744 tablet_service.cc:1514] Processing DeleteTablet for tablet d734ddb17ce045a686625d0396f8a06a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:27:31 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40378
I20260927 21:27:31.228549 7866 raft_consensus.cc:2229] T d734ddb17ce045a686625d0396f8a06a P 4ff9eaae578a43f0aa15395e349d3117 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:27:31.229115 7866 raft_consensus.cc:2258] T d734ddb17ce045a686625d0396f8a06a P 4ff9eaae578a43f0aa15395e349d3117 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:27:31.230307 7458 tablet_service.cc:1514] Processing DeleteTablet for tablet d734ddb17ce045a686625d0396f8a06a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:27:31 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42278
I20260927 21:27:31.232466 7866 ts_tablet_manager.cc:1905] T d734ddb17ce045a686625d0396f8a06a P 4ff9eaae578a43f0aa15395e349d3117: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:27:31.240995 7866 ts_tablet_manager.cc:1918] T d734ddb17ce045a686625d0396f8a06a P 4ff9eaae578a43f0aa15395e349d3117: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:27:31.241176 7866 log.cc:1196] T d734ddb17ce045a686625d0396f8a06a P 4ff9eaae578a43f0aa15395e349d3117: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/wal/wals/d734ddb17ce045a686625d0396f8a06a
I20260927 21:27:31.241878 7866 ts_tablet_manager.cc:1939] T d734ddb17ce045a686625d0396f8a06a P 4ff9eaae578a43f0aa15395e349d3117: Deleting consensus metadata
I20260927 21:27:31.244191 6542 catalog_manager.cc:4841] TS 4ff9eaae578a43f0aa15395e349d3117 (127.0.97.2:38203): tablet d734ddb17ce045a686625d0396f8a06a (table test-table [id=e9236fb62cb441c7a57c7292782983d5]) successfully deleted
I20260927 21:27:31.254951 7868 tablet_replica.cc:330] T d734ddb17ce045a686625d0396f8a06a P 7f60711999224fc782e173cc4e4af402: stopping tablet replica
I20260927 21:27:31.255468 7868 raft_consensus.cc:2229] T d734ddb17ce045a686625d0396f8a06a P 7f60711999224fc782e173cc4e4af402 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:27:31.256071 7868 raft_consensus.cc:2258] T d734ddb17ce045a686625d0396f8a06a P 7f60711999224fc782e173cc4e4af402 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:27:31.259270 7867 tablet_replica.cc:330] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077: stopping tablet replica
I20260927 21:27:31.259325 7868 ts_tablet_manager.cc:1905] T d734ddb17ce045a686625d0396f8a06a P 7f60711999224fc782e173cc4e4af402: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:27:31.259752 7867 raft_consensus.cc:2229] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:27:31.260484 7867 raft_consensus.cc:2258] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:27:31.262835 7867 ts_tablet_manager.cc:1905] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:27:31.265556 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 6525
I20260927 21:27:31.267748 6587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:31.276744 7867 ts_tablet_manager.cc:1918] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:27:31.278432 7868 ts_tablet_manager.cc:1918] T d734ddb17ce045a686625d0396f8a06a P 7f60711999224fc782e173cc4e4af402: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:27:31.278731 7868 log.cc:1196] T d734ddb17ce045a686625d0396f8a06a P 7f60711999224fc782e173cc4e4af402: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/wal/wals/d734ddb17ce045a686625d0396f8a06a
I20260927 21:27:31.279251 7868 ts_tablet_manager.cc:1939] T d734ddb17ce045a686625d0396f8a06a P 7f60711999224fc782e173cc4e4af402: Deleting consensus metadata
I20260927 21:27:31.280607 7867 log.cc:1196] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/wal/wals/d734ddb17ce045a686625d0396f8a06a
I20260927 21:27:31.281587 7867 ts_tablet_manager.cc:1939] T d734ddb17ce045a686625d0396f8a06a P 277d8b80cca4431c9917b2ea4570c077: Deleting consensus metadata
I20260927 21:27:31.288515 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:31.231852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52666 (local address 127.0.97.61:34259)
0927 21:27:31.232026 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:31.232035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:31.254343 (+ 22308us) server_negotiation.cc:407] Connection header received
0927 21:27:31.254819 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:31.254824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:31.254910 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:31.255017 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:31.269093 (+ 14076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.270324 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.272737 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.273403 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.273510 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:31.280787 (+ 7277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:31.280796 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:31.280802 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:31.280850 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:31.284694 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:31.285924 (+ 1230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:31.285930 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:31.285934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:31.286102 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:31.286678 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:31.286683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:31.286687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:31.287036 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:31.287237 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:31.287911 (+ 674us) server_negotiation.cc:299] Negotiation successful
0927 21:27:31.288151 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:27:31.305583 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:31.254532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43038 (local address 127.0.97.60:44991)
0927 21:27:31.254721 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:31.254726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:31.291279 (+ 36553us) server_negotiation.cc:407] Connection header received
0927 21:27:31.291512 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:31.291519 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:31.291596 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:31.291660 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:31.293688 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.294876 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.296835 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.297446 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.297563 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:31.300167 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:31.300173 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:31.300178 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:31.300221 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:31.302893 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:31.303541 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:31.303547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:31.303550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:31.303678 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:31.304041 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:31.304047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:31.304050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:31.304321 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:31.304624 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:31.304912 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 21:27:31.305119 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 21:27:31.447954 6544 catalog_manager.cc:4841] TS 277d8b80cca4431c9917b2ea4570c077 (127.0.97.3:33731): tablet d734ddb17ce045a686625d0396f8a06a (table test-table [id=e9236fb62cb441c7a57c7292782983d5]) successfully deleted
W20260927 21:27:31.448232 6544 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.97.3:33731, user_credentials={real_user=slave}} blocked reactor thread for 164001us
I20260927 21:27:31.448618 6544 catalog_manager.cc:4841] TS 7f60711999224fc782e173cc4e4af402 (127.0.97.1:33945): tablet d734ddb17ce045a686625d0396f8a06a (table test-table [id=e9236fb62cb441c7a57c7292782983d5]) successfully deleted
I20260927 21:27:31.448855 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 6525
I20260927 21:27:31.510334 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:31.477750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39882 (local address 127.0.97.61:34259)
0927 21:27:31.478040 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:31.478051 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:31.493738 (+ 15687us) server_negotiation.cc:407] Connection header received
0927 21:27:31.494360 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:31.494368 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:31.494451 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:31.494524 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:31.497031 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.497954 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.499301 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.499743 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.499893 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:31.503933 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:31.503942 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:31.503947 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:31.503995 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:31.507175 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:31.508089 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:31.508097 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:31.508100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:31.508228 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:31.508642 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:31.508650 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:31.508655 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:31.509095 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:31.509338 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:31.509756 (+ 418us) server_negotiation.cc:299] Negotiation successful
0927 21:27:31.509991 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:27:31.511884 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:31.494025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53794 (local address 127.0.97.60:44991)
0927 21:27:31.494239 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:31.494245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:31.494266 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:27:31.494429 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:31.494434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:31.494509 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:31.494619 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:31.496385 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.497259 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.499328 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:31.500043 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:31.500193 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:31.504732 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:31.504741 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:31.504747 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:31.504791 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:31.508440 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:31.509308 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:31.509315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:31.509319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:31.509468 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:31.509945 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:31.509950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:31.509954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:31.510377 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:31.510677 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:31.511301 (+ 624us) server_negotiation.cc:299] Negotiation successful
0927 21:27:31.511526 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 21:27:32.204718 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:32.203832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.3:38355)
0927 21:27:32.204380 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:32.204532 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":285,"threads_started":1}
W20260927 21:27:32.205302 7810 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39737 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
I20260927 21:27:32.214027 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:32.213141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.1:47167)
0927 21:27:32.213683 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:32.213800 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":316,"threads_started":1}
I20260927 21:27:32.214402 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:32.213715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.2:48891)
0927 21:27:32.214117 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:32.214236 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":246,"threads_started":1}
W20260927 21:27:32.214432 7524 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39737 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
W20260927 21:27:32.214805 7669 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39737 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
I20260927 21:27:32.725888 7880 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a978f7733b8c466f9f9ef4d1acefb6d2)
I20260927 21:27:32.726013 7881 raft_consensus.cc:482] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a978f7733b8c466f9f9ef4d1acefb6d2)
I20260927 21:27:32.726239 7880 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: false } }
I20260927 21:27:32.726320 7881 raft_consensus.cc:504] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: false } }
I20260927 21:27:32.727895 7881 leader_election.cc:288] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a978f7733b8c466f9f9ef4d1acefb6d2 (127.0.97.62:39737), 9f5af1173bb24003b4cac6a9beac8b91 (127.0.97.61:34259)
I20260927 21:27:32.730589 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:32.727963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.0.1:33518)
0927 21:27:32.730101 (+ 2138us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:32.730301 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1961,"threads_started":1}
I20260927 21:27:32.734819 7880 leader_election.cc:288] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a978f7733b8c466f9f9ef4d1acefb6d2 (127.0.97.62:39737), da163c81fef446f9bc4b8a31a02b413e (127.0.97.60:44991)
I20260927 21:27:32.741657 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:32.727988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.0.1:33526)
0927 21:27:32.741113 (+ 13125us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:32.741311 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12970,"thread_start_us":12820,"threads_started":1}
W20260927 21:27:32.742245 7141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
W20260927 21:27:32.742694 7329 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
I20260927 21:27:32.749699 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:32.728024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44991 (local address 127.0.0.1:53796)
0927 21:27:32.732399 (+ 4375us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:32.732422 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:32.732622 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:32.742095 (+ 9473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:32.742100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:32.742136 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:32.742491 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:32.742505 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:32.744220 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:32.744226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:32.746390 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:32.746407 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:32.748499 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:32.748506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:32.748727 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:32.748763 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 21:27:32.748863 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 21:27:32.749038 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4250,"thread_start_us":1756,"threads_started":1}
I20260927 21:27:32.749858 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:32.728509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53796 (local address 127.0.97.60:44991)
0927 21:27:32.741168 (+ 12659us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:32.741174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:32.741194 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:27:32.741269 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:32.741275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:32.741362 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:32.741507 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:32.742693 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:32.744067 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:32.746591 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:32.748124 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:32.748258 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:32.748382 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:32.749327 (+ 945us) server_negotiation.cc:299] Negotiation successful
0927 21:27:32.749475 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12545,"thread_start_us":12395,"threads_started":1}
I20260927 21:27:32.750202 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:32.728394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34259 (local address 127.0.0.1:39898)
0927 21:27:32.742075 (+ 13681us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:32.742093 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:32.742212 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:32.742772 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:32.742778 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:32.742824 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:32.743263 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:32.743275 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:32.744885 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:32.744890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:32.747037 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:32.747050 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:32.749383 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:32.749389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:32.749571 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:32.749639 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 21:27:32.749725 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 21:27:32.749878 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13565,"thread_start_us":13411,"threads_started":1}
I20260927 21:27:32.751047 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:32.728724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39898 (local address 127.0.97.61:34259)
0927 21:27:32.731892 (+ 3168us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:32.731902 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:32.742185 (+ 10283us) server_negotiation.cc:407] Connection header received
0927 21:27:32.742343 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:32.742354 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:32.742421 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:32.742541 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:32.743503 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:32.744704 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:32.747214 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:32.748933 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:32.749100 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:32.750224 (+ 1124us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:32.750287 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 21:27:32.750421 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2990,"thread_start_us":2815,"threads_started":1}
I20260927 21:27:32.750907 7354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f5af1173bb24003b4cac6a9beac8b91" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "da163c81fef446f9bc4b8a31a02b413e" is_pre_election: true
I20260927 21:27:32.751361 7354 raft_consensus.cc:2454] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f5af1173bb24003b4cac6a9beac8b91 in term 1.
I20260927 21:27:32.751612 7166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "da163c81fef446f9bc4b8a31a02b413e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9f5af1173bb24003b4cac6a9beac8b91" is_pre_election: true
I20260927 21:27:32.752149 7166 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da163c81fef446f9bc4b8a31a02b413e in term 1.
I20260927 21:27:32.752161 7143 leader_election.cc:302] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f5af1173bb24003b4cac6a9beac8b91, da163c81fef446f9bc4b8a31a02b413e; no voters:
I20260927 21:27:32.752622 7880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 21:27:32.752758 7880 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a978f7733b8c466f9f9ef4d1acefb6d2)
I20260927 21:27:32.752725 7331 leader_election.cc:302] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f5af1173bb24003b4cac6a9beac8b91, da163c81fef446f9bc4b8a31a02b413e; no voters:
I20260927 21:27:32.752882 7880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:27:32.752964 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:32.752507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.0.1:33536)
0927 21:27:32.752728 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:32.752823 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 21:27:32.753121 7881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 21:27:32.753269 7881 raft_consensus.cc:482] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [term 1 FOLLOWER]: Starting leader election (detected failure of leader a978f7733b8c466f9f9ef4d1acefb6d2)
I20260927 21:27:32.753381 7881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [term 1 FOLLOWER]: Advancing to term 2
W20260927 21:27:32.753135 7329 leader_election.cc:334] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a978f7733b8c466f9f9ef4d1acefb6d2 (127.0.97.62:39737): Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
I20260927 21:27:32.753791 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:32.753317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.0.1:33546)
0927 21:27:32.753505 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:32.753591 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260927 21:27:32.754024 7141 leader_election.cc:334] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a978f7733b8c466f9f9ef4d1acefb6d2 (127.0.97.62:39737): Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
I20260927 21:27:32.755605 7880 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: false } }
I20260927 21:27:32.755749 7881 raft_consensus.cc:504] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: false } }
I20260927 21:27:32.756357 7880 leader_election.cc:288] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [CANDIDATE]: Term 2 election: Requested vote from peers a978f7733b8c466f9f9ef4d1acefb6d2 (127.0.97.62:39737), da163c81fef446f9bc4b8a31a02b413e (127.0.97.60:44991)
I20260927 21:27:32.756464 7881 leader_election.cc:288] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [CANDIDATE]: Term 2 election: Requested vote from peers a978f7733b8c466f9f9ef4d1acefb6d2 (127.0.97.62:39737), 9f5af1173bb24003b4cac6a9beac8b91 (127.0.97.61:34259)
I20260927 21:27:32.756969 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:32.756484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.0.1:33560)
0927 21:27:32.756673 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:32.756822 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 21:27:32.757035 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:32.756521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.0.1:33564)
0927 21:27:32.756747 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:32.756861 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260927 21:27:32.757042 7354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f5af1173bb24003b4cac6a9beac8b91" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "da163c81fef446f9bc4b8a31a02b413e"
I20260927 21:27:32.757054 7166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "da163c81fef446f9bc4b8a31a02b413e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9f5af1173bb24003b4cac6a9beac8b91"
I20260927 21:27:32.757388 7354 raft_consensus.cc:2379] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f5af1173bb24003b4cac6a9beac8b91 in current term 2: Already voted for candidate da163c81fef446f9bc4b8a31a02b413e in this term.
I20260927 21:27:32.757437 7166 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate da163c81fef446f9bc4b8a31a02b413e in current term 2: Already voted for candidate 9f5af1173bb24003b4cac6a9beac8b91 in this term.
I20260927 21:27:32.758185 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:32.757771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.0.1:33566)
0927 21:27:32.757968 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:32.758048 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260927 21:27:32.758203 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:32.757811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.0.1:33570)
0927 21:27:32.758000 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:32.758076 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260927 21:27:32.758368 7329 leader_election.cc:334] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a978f7733b8c466f9f9ef4d1acefb6d2 (127.0.97.62:39737): Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
I20260927 21:27:32.758466 7329 leader_election.cc:302] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da163c81fef446f9bc4b8a31a02b413e; no voters: 9f5af1173bb24003b4cac6a9beac8b91, a978f7733b8c466f9f9ef4d1acefb6d2
W20260927 21:27:32.758481 7141 leader_election.cc:334] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a978f7733b8c466f9f9ef4d1acefb6d2 (127.0.97.62:39737): Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
I20260927 21:27:32.758597 7141 leader_election.cc:302] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f5af1173bb24003b4cac6a9beac8b91; no voters: a978f7733b8c466f9f9ef4d1acefb6d2, da163c81fef446f9bc4b8a31a02b413e
I20260927 21:27:32.758749 7881 raft_consensus.cc:2726] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 21:27:32.758916 7880 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 21:27:33.207180 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:33.206305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.3:47391)
0927 21:27:33.206785 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:33.206907 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":241,"threads_started":1}
I20260927 21:27:33.216049 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:33.215178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.1:35391)
0927 21:27:33.215760 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:33.215869 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":365,"threads_started":1}
I20260927 21:27:33.216084 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:33.215430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.2:58647)
0927 21:27:33.215843 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:33.215918 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":220,"threads_started":1}
I20260927 21:27:34.209400 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.208361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.3:36619)
0927 21:27:34.209008 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.209162 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":250,"threads_started":1}
I20260927 21:27:34.217736 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.216898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.2:40587)
0927 21:27:34.217404 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.217521 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":236,"threads_started":1}
I20260927 21:27:34.217825 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.217079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.1:42917)
0927 21:27:34.217503 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.217588 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":256,"threads_started":1}
I20260927 21:27:34.458567 7896 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:27:34.458820 7896 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: false } }
I20260927 21:27:34.459666 7896 leader_election.cc:288] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a978f7733b8c466f9f9ef4d1acefb6d2 (127.0.97.62:39737), da163c81fef446f9bc4b8a31a02b413e (127.0.97.60:44991)
I20260927 21:27:34.460379 7354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f5af1173bb24003b4cac6a9beac8b91" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "da163c81fef446f9bc4b8a31a02b413e" is_pre_election: true
I20260927 21:27:34.460649 7354 raft_consensus.cc:2454] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f5af1173bb24003b4cac6a9beac8b91 in term 2.
I20260927 21:27:34.460861 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.459912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.0.1:33830)
0927 21:27:34.460471 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.460596 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":283,"threads_started":1}
I20260927 21:27:34.461158 7143 leader_election.cc:302] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f5af1173bb24003b4cac6a9beac8b91, da163c81fef446f9bc4b8a31a02b413e; no voters:
I20260927 21:27:34.461578 7896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 21:27:34.461764 7896 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:27:34.461939 7896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 2 FOLLOWER]: Advancing to term 3
I20260927 21:27:34.462456 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.462103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.0.1:33832)
0927 21:27:34.462251 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.462344 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 21:27:34.462631 7141 leader_election.cc:334] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a978f7733b8c466f9f9ef4d1acefb6d2 (127.0.97.62:39737): Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
I20260927 21:27:34.464340 7896 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: false } }
I20260927 21:27:34.465034 7896 leader_election.cc:288] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [CANDIDATE]: Term 3 election: Requested vote from peers a978f7733b8c466f9f9ef4d1acefb6d2 (127.0.97.62:39737), da163c81fef446f9bc4b8a31a02b413e (127.0.97.60:44991)
I20260927 21:27:34.465639 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.465172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.0.1:33844)
0927 21:27:34.465341 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.465434 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 21:27:34.465672 7354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f5af1173bb24003b4cac6a9beac8b91" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "da163c81fef446f9bc4b8a31a02b413e"
I20260927 21:27:34.465987 7354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [term 2 FOLLOWER]: Advancing to term 3
I20260927 21:27:34.466687 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.466306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.0.1:33856)
0927 21:27:34.466469 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.466553 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 21:27:34.467335 7141 leader_election.cc:334] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a978f7733b8c466f9f9ef4d1acefb6d2 (127.0.97.62:39737): Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
I20260927 21:27:34.468745 7354 raft_consensus.cc:2454] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f5af1173bb24003b4cac6a9beac8b91 in term 3.
I20260927 21:27:34.469201 7143 leader_election.cc:302] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9f5af1173bb24003b4cac6a9beac8b91, da163c81fef446f9bc4b8a31a02b413e; no voters: a978f7733b8c466f9f9ef4d1acefb6d2
I20260927 21:27:34.469533 7896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 21:27:34.470059 7896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [term 3 LEADER]: Becoming Leader. State: Replica: 9f5af1173bb24003b4cac6a9beac8b91, State: Running, Role: LEADER
I20260927 21:27:34.470492 7896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: false } }
I20260927 21:27:34.472935 7899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f5af1173bb24003b4cac6a9beac8b91. Latest consensus state: current_term: 3 leader_uuid: "9f5af1173bb24003b4cac6a9beac8b91" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: false } } }
I20260927 21:27:34.473264 7899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:34.474076 7901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:27:34.477964 7901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:27:34.478940 7901 catalog_manager.cc:1259] Loaded cluster ID: 700f24069d884b139152c6fa230b5dd4
I20260927 21:27:34.479063 7901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:27:34.480392 7901 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:27:34.481287 7901 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91: Loaded TSK: 0
I20260927 21:27:34.482251 7901 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:27:34.554366 7156 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:27:34.561972 7156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:27:34.566623 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.566164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.0.1:33882)
0927 21:27:34.566383 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.566466 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 21:27:34.566676 7354 raft_consensus.cc:1261] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [term 3 FOLLOWER]: Refusing update from remote peer 9f5af1173bb24003b4cac6a9beac8b91: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 21:27:34.567507 7899 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 21:27:34.567811 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.567411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.0.1:33886)
0927 21:27:34.567597 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.567675 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 21:27:34.567991 7141 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 -> Peer a978f7733b8c466f9f9ef4d1acefb6d2 (127.0.97.62:39737): Couldn't send request to peer a978f7733b8c466f9f9ef4d1acefb6d2. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 21:27:34.570365 7903 sys_catalog.cc:455] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f5af1173bb24003b4cac6a9beac8b91. Latest consensus state: current_term: 3 leader_uuid: "9f5af1173bb24003b4cac6a9beac8b91" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: false } } }
I20260927 21:27:34.570670 7903 sys_catalog.cc:458] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:27:34.572141 7896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "9f5af1173bb24003b4cac6a9beac8b91" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: false } } }
I20260927 21:27:34.572525 7896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:34.573072 7903 sys_catalog.cc:455] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "9f5af1173bb24003b4cac6a9beac8b91" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a978f7733b8c466f9f9ef4d1acefb6d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39737 } } peers { permanent_uuid: "9f5af1173bb24003b4cac6a9beac8b91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34259 } attrs { promote: false } } peers { permanent_uuid: "da163c81fef446f9bc4b8a31a02b413e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44991 } attrs { promote: false } } }
I20260927 21:27:34.573348 7903 sys_catalog.cc:458] T 00000000000000000000000000000000 P da163c81fef446f9bc4b8a31a02b413e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:27:34.593288 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.581173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38203 (local address 127.0.0.1:42870)
0927 21:27:34.581715 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.581735 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:34.581860 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:34.584090 (+ 2230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:34.584099 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:34.584135 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:34.584587 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:34.584603 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:34.586611 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:34.586620 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:34.589553 (+ 2933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:34.589580 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:34.592022 (+ 2442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:34.592033 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:34.592333 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:34.592386 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 21:27:34.592497 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 21:27:34.592730 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":580,"threads_started":1}
I20260927 21:27:34.594312 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.581636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42870 (local address 127.0.97.2:38203)
0927 21:27:34.583473 (+ 1837us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:34.583483 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:34.583508 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:27:34.583597 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:34.583605 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:34.583704 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:34.583822 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:34.584875 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:34.586377 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:34.589869 (+ 3492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:34.591550 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:34.591720 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:34.591874 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:34.593636 (+ 1762us) server_negotiation.cc:299] Negotiation successful
0927 21:27:34.593851 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1671,"thread_start_us":1545,"threads_started":1}
I20260927 21:27:34.595327 7603 tablet_service.cc:1467] Processing CreateTablet for tablet b2fa80d2ade2498b82237126f6255a91 (DEFAULT_TABLE table=test-table [id=b5f51f12b9954ddbbccd780ebbb884fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:27:34.595970 7603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2fa80d2ade2498b82237126f6255a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:34.602397 7914 tablet_bootstrap.cc:492] T b2fa80d2ade2498b82237126f6255a91 P 4ff9eaae578a43f0aa15395e349d3117: Bootstrap starting.
I20260927 21:27:34.603430 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.580952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33731 (local address 127.0.0.1:58824)
0927 21:27:34.581115 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.581138 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:34.581284 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:34.597326 (+ 16042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:34.597331 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:34.597359 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:34.597642 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:34.597657 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:34.599019 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:34.599024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:34.601131 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:34.601144 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:34.602613 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:34.602619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:34.602798 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:34.602836 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 21:27:34.602906 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 21:27:34.603035 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":62}
I20260927 21:27:34.604492 7914 tablet_bootstrap.cc:654] T b2fa80d2ade2498b82237126f6255a91 P 4ff9eaae578a43f0aa15395e349d3117: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:34.604648 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.581031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58824 (local address 127.0.97.3:33731)
0927 21:27:34.597048 (+ 16017us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:34.597053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:34.597069 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:27:34.597124 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:34.597128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:34.597183 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:34.597313 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:34.597845 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:34.598897 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:34.601294 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:34.602376 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:34.603882 (+ 1506us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:34.603990 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:34.604042 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 21:27:34.604171 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15904,"thread_start_us":16445,"threads_started":1}
I20260927 21:27:34.605011 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.583335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33945 (local address 127.0.0.1:46902)
0927 21:27:34.595148 (+ 11813us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.595173 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:34.595311 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:34.598328 (+ 3017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:34.598333 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:34.598370 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:34.598523 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:34.598533 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:34.599726 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:34.599732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:34.601049 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:34.601063 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:34.604137 (+ 3074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:34.604143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:34.604393 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:34.604442 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 21:27:34.604515 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 21:27:34.604669 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11685,"mutex_wait_us":26,"thread_start_us":20643,"threads_started":1}
I20260927 21:27:34.605396 7744 tablet_service.cc:1467] Processing CreateTablet for tablet b2fa80d2ade2498b82237126f6255a91 (DEFAULT_TABLE table=test-table [id=b5f51f12b9954ddbbccd780ebbb884fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:27:34.605846 7744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2fa80d2ade2498b82237126f6255a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:34.605906 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.583485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46902 (local address 127.0.97.1:33945)
0927 21:27:34.598007 (+ 14522us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:34.598015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:34.598034 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:27:34.598099 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:34.598104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:34.598181 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:34.598650 (+ 469us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:34.598827 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:34.599588 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:34.601227 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:34.603956 (+ 2729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:34.605243 (+ 1287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:34.605360 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:34.605417 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 21:27:34.605548 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14392,"thread_start_us":14281,"threads_started":1}
I20260927 21:27:34.606626 7458 tablet_service.cc:1467] Processing CreateTablet for tablet b2fa80d2ade2498b82237126f6255a91 (DEFAULT_TABLE table=test-table [id=b5f51f12b9954ddbbccd780ebbb884fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:27:34.606958 7914 tablet_bootstrap.cc:492] T b2fa80d2ade2498b82237126f6255a91 P 4ff9eaae578a43f0aa15395e349d3117: No bootstrap required, opened a new log
I20260927 21:27:34.607018 7458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2fa80d2ade2498b82237126f6255a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:34.607098 7914 ts_tablet_manager.cc:1397] T b2fa80d2ade2498b82237126f6255a91 P 4ff9eaae578a43f0aa15395e349d3117: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 21:27:34.608031 7914 raft_consensus.cc:348] T b2fa80d2ade2498b82237126f6255a91 P 4ff9eaae578a43f0aa15395e349d3117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277d8b80cca4431c9917b2ea4570c077" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33731 } } peers { permanent_uuid: "4ff9eaae578a43f0aa15395e349d3117" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38203 } } peers { permanent_uuid: "7f60711999224fc782e173cc4e4af402" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33945 } }
I20260927 21:27:34.608263 7914 raft_consensus.cc:374] T b2fa80d2ade2498b82237126f6255a91 P 4ff9eaae578a43f0aa15395e349d3117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:34.608349 7914 raft_consensus.cc:729] T b2fa80d2ade2498b82237126f6255a91 P 4ff9eaae578a43f0aa15395e349d3117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ff9eaae578a43f0aa15395e349d3117, State: Initialized, Role: FOLLOWER
I20260927 21:27:34.608604 7914 consensus_queue.cc:260] T b2fa80d2ade2498b82237126f6255a91 P 4ff9eaae578a43f0aa15395e349d3117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277d8b80cca4431c9917b2ea4570c077" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33731 } } peers { permanent_uuid: "4ff9eaae578a43f0aa15395e349d3117" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38203 } } peers { permanent_uuid: "7f60711999224fc782e173cc4e4af402" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33945 } }
I20260927 21:27:34.609748 7914 ts_tablet_manager.cc:1428] T b2fa80d2ade2498b82237126f6255a91 P 4ff9eaae578a43f0aa15395e349d3117: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:27:34.610733 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.610337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.2:48845)
0927 21:27:34.610484 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.610577 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 21:27:34.613670 7670 heartbeater.cc:503] Master 127.0.97.61:34259 was elected leader, sending a full tablet report...
I20260927 21:27:34.613734 7917 tablet_bootstrap.cc:492] T b2fa80d2ade2498b82237126f6255a91 P 277d8b80cca4431c9917b2ea4570c077: Bootstrap starting.
I20260927 21:27:34.614488 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.614023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.2:41035)
0927 21:27:34.614228 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.614328 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 21:27:34.615262 7918 tablet_bootstrap.cc:492] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402: Bootstrap starting.
I20260927 21:27:34.615914 7917 tablet_bootstrap.cc:654] T b2fa80d2ade2498b82237126f6255a91 P 277d8b80cca4431c9917b2ea4570c077: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:34.617240 7918 tablet_bootstrap.cc:654] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:34.618459 7917 tablet_bootstrap.cc:492] T b2fa80d2ade2498b82237126f6255a91 P 277d8b80cca4431c9917b2ea4570c077: No bootstrap required, opened a new log
I20260927 21:27:34.618623 7917 ts_tablet_manager.cc:1397] T b2fa80d2ade2498b82237126f6255a91 P 277d8b80cca4431c9917b2ea4570c077: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 21:27:34.619481 7918 tablet_bootstrap.cc:492] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402: No bootstrap required, opened a new log
I20260927 21:27:34.619488 7917 raft_consensus.cc:348] T b2fa80d2ade2498b82237126f6255a91 P 277d8b80cca4431c9917b2ea4570c077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277d8b80cca4431c9917b2ea4570c077" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33731 } } peers { permanent_uuid: "4ff9eaae578a43f0aa15395e349d3117" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38203 } } peers { permanent_uuid: "7f60711999224fc782e173cc4e4af402" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33945 } }
I20260927 21:27:34.619663 7918 ts_tablet_manager.cc:1397] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260927 21:27:34.619766 7917 raft_consensus.cc:374] T b2fa80d2ade2498b82237126f6255a91 P 277d8b80cca4431c9917b2ea4570c077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:34.619915 7917 raft_consensus.cc:729] T b2fa80d2ade2498b82237126f6255a91 P 277d8b80cca4431c9917b2ea4570c077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 277d8b80cca4431c9917b2ea4570c077, State: Initialized, Role: FOLLOWER
I20260927 21:27:34.620209 7917 consensus_queue.cc:260] T b2fa80d2ade2498b82237126f6255a91 P 277d8b80cca4431c9917b2ea4570c077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277d8b80cca4431c9917b2ea4570c077" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33731 } } peers { permanent_uuid: "4ff9eaae578a43f0aa15395e349d3117" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38203 } } peers { permanent_uuid: "7f60711999224fc782e173cc4e4af402" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33945 } }
I20260927 21:27:34.620460 7918 raft_consensus.cc:348] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277d8b80cca4431c9917b2ea4570c077" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33731 } } peers { permanent_uuid: "4ff9eaae578a43f0aa15395e349d3117" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38203 } } peers { permanent_uuid: "7f60711999224fc782e173cc4e4af402" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33945 } }
I20260927 21:27:34.620751 7918 raft_consensus.cc:374] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:34.620841 7918 raft_consensus.cc:729] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f60711999224fc782e173cc4e4af402, State: Initialized, Role: FOLLOWER
I20260927 21:27:34.621122 7918 consensus_queue.cc:260] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277d8b80cca4431c9917b2ea4570c077" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33731 } } peers { permanent_uuid: "4ff9eaae578a43f0aa15395e349d3117" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38203 } } peers { permanent_uuid: "7f60711999224fc782e173cc4e4af402" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33945 } }
I20260927 21:27:34.621299 7917 ts_tablet_manager.cc:1428] T b2fa80d2ade2498b82237126f6255a91 P 277d8b80cca4431c9917b2ea4570c077: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:27:34.621978 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.621514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.3:57715)
0927 21:27:34.621723 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.621817 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 21:27:34.622619 7918 ts_tablet_manager.cc:1428] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 21:27:34.623556 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.623169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.1:36367)
0927 21:27:34.623297 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.623410 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 21:27:34.624310 7811 heartbeater.cc:503] Master 127.0.97.61:34259 was elected leader, sending a full tablet report...
I20260927 21:27:34.625442 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.625054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.3:39377)
0927 21:27:34.625242 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.625324 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 21:27:34.626247 7525 heartbeater.cc:503] Master 127.0.97.61:34259 was elected leader, sending a full tablet report...
I20260927 21:27:34.626921 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.626504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.1:51511)
0927 21:27:34.626656 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.626764 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 21:27:34.785090 7677 tablet.cc:2369] T b2fa80d2ade2498b82237126f6255a91 P 4ff9eaae578a43f0aa15395e349d3117: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:27:34.873080 7922 raft_consensus.cc:482] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:27:34.873384 7922 raft_consensus.cc:504] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277d8b80cca4431c9917b2ea4570c077" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33731 } } peers { permanent_uuid: "4ff9eaae578a43f0aa15395e349d3117" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38203 } } peers { permanent_uuid: "7f60711999224fc782e173cc4e4af402" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33945 } }
I20260927 21:27:34.875346 7922 leader_election.cc:288] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 277d8b80cca4431c9917b2ea4570c077 (127.0.97.3:33731), 4ff9eaae578a43f0aa15395e349d3117 (127.0.97.2:38203)
I20260927 21:27:34.883610 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.876083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53463 (local address 127.0.97.2:38203)
0927 21:27:34.876265 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:34.876271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:34.876380 (+ 109us) server_negotiation.cc:407] Connection header received
0927 21:27:34.876555 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:34.876559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:34.876615 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:34.876691 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:34.877357 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:34.878384 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:34.881203 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:34.882475 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:34.882618 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:34.882725 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:34.883209 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 21:27:34.883349 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 21:27:34.883616 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.875585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50955 (local address 127.0.97.3:33731)
0927 21:27:34.875812 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:34.875819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:34.876149 (+ 330us) server_negotiation.cc:407] Connection header received
0927 21:27:34.876227 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:34.876233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:34.876317 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:34.876402 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:34.877167 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:34.878281 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:34.881203 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:34.882492 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:34.882618 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:34.882725 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:34.883209 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 21:27:34.883348 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 21:27:34.883692 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.875489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33731 (local address 127.0.97.1:50955)
0927 21:27:34.875811 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.875831 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:34.876003 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:34.876572 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:34.876575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:34.876597 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:34.877011 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:34.877024 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:34.878459 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:34.878467 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:34.881011 (+ 2544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:34.881024 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:34.882740 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:34.882754 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:34.882966 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:34.883017 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 21:27:34.883120 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 21:27:34.883322 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":194}
I20260927 21:27:34.883742 7923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.875582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38203 (local address 127.0.97.1:53463)
0927 21:27:34.876265 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.876281 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:34.876413 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:34.876735 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:34.876739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:34.876756 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:34.877202 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:34.877211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:34.878502 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:34.878509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:34.881009 (+ 2500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:34.881024 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:34.882766 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:34.882771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:34.882960 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:34.883029 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 21:27:34.883120 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 21:27:34.883322 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":249,"thread_start_us":626,"threads_started":1}
I20260927 21:27:34.884645 7764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2fa80d2ade2498b82237126f6255a91" candidate_uuid: "7f60711999224fc782e173cc4e4af402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "277d8b80cca4431c9917b2ea4570c077" is_pre_election: true
I20260927 21:27:34.884763 7623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2fa80d2ade2498b82237126f6255a91" candidate_uuid: "7f60711999224fc782e173cc4e4af402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ff9eaae578a43f0aa15395e349d3117" is_pre_election: true
I20260927 21:27:34.885044 7623 raft_consensus.cc:2454] T b2fa80d2ade2498b82237126f6255a91 P 4ff9eaae578a43f0aa15395e349d3117 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f60711999224fc782e173cc4e4af402 in term 0.
I20260927 21:27:34.885077 7764 raft_consensus.cc:2454] T b2fa80d2ade2498b82237126f6255a91 P 277d8b80cca4431c9917b2ea4570c077 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f60711999224fc782e173cc4e4af402 in term 0.
I20260927 21:27:34.885591 7412 leader_election.cc:302] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ff9eaae578a43f0aa15395e349d3117, 7f60711999224fc782e173cc4e4af402; no voters:
I20260927 21:27:34.886047 7922 raft_consensus.cc:2781] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:27:34.886183 7922 raft_consensus.cc:482] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:27:34.886276 7922 raft_consensus.cc:3037] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:34.888437 7922 raft_consensus.cc:504] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277d8b80cca4431c9917b2ea4570c077" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33731 } } peers { permanent_uuid: "4ff9eaae578a43f0aa15395e349d3117" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38203 } } peers { permanent_uuid: "7f60711999224fc782e173cc4e4af402" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33945 } }
I20260927 21:27:34.889104 7922 leader_election.cc:288] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402 [CANDIDATE]: Term 1 election: Requested vote from peers 277d8b80cca4431c9917b2ea4570c077 (127.0.97.3:33731), 4ff9eaae578a43f0aa15395e349d3117 (127.0.97.2:38203)
I20260927 21:27:34.889596 7764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2fa80d2ade2498b82237126f6255a91" candidate_uuid: "7f60711999224fc782e173cc4e4af402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "277d8b80cca4431c9917b2ea4570c077"
I20260927 21:27:34.889699 7623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2fa80d2ade2498b82237126f6255a91" candidate_uuid: "7f60711999224fc782e173cc4e4af402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ff9eaae578a43f0aa15395e349d3117"
I20260927 21:27:34.889859 7764 raft_consensus.cc:3037] T b2fa80d2ade2498b82237126f6255a91 P 277d8b80cca4431c9917b2ea4570c077 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:34.889889 7623 raft_consensus.cc:3037] T b2fa80d2ade2498b82237126f6255a91 P 4ff9eaae578a43f0aa15395e349d3117 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:34.892868 7623 raft_consensus.cc:2454] T b2fa80d2ade2498b82237126f6255a91 P 4ff9eaae578a43f0aa15395e349d3117 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f60711999224fc782e173cc4e4af402 in term 1.
I20260927 21:27:34.892879 7764 raft_consensus.cc:2454] T b2fa80d2ade2498b82237126f6255a91 P 277d8b80cca4431c9917b2ea4570c077 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f60711999224fc782e173cc4e4af402 in term 1.
I20260927 21:27:34.893321 7412 leader_election.cc:302] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 277d8b80cca4431c9917b2ea4570c077, 4ff9eaae578a43f0aa15395e349d3117, 7f60711999224fc782e173cc4e4af402; no voters:
I20260927 21:27:34.893730 7922 raft_consensus.cc:2781] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:27:34.894276 7922 raft_consensus.cc:686] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402 [term 1 LEADER]: Becoming Leader. State: Replica: 7f60711999224fc782e173cc4e4af402, State: Running, Role: LEADER
I20260927 21:27:34.894936 7922 consensus_queue.cc:237] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277d8b80cca4431c9917b2ea4570c077" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33731 } } peers { permanent_uuid: "4ff9eaae578a43f0aa15395e349d3117" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38203 } } peers { permanent_uuid: "7f60711999224fc782e173cc4e4af402" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33945 } }
I20260927 21:27:34.897701 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.897192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.1:36231)
0927 21:27:34.897377 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.897484 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 21:27:34.898746 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.898423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.1:39209)
0927 21:27:34.898580 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.898642 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 21:27:34.898972 7156 catalog_manager.cc:5495] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f60711999224fc782e173cc4e4af402 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "7f60711999224fc782e173cc4e4af402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277d8b80cca4431c9917b2ea4570c077" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ff9eaae578a43f0aa15395e349d3117" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f60711999224fc782e173cc4e4af402" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33945 } health_report { overall_health: HEALTHY } } }
I20260927 21:27:34.958416 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.953391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46918 (local address 127.0.97.1:33945)
0927 21:27:34.953582 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:34.953587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:34.953605 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:27:34.953717 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:34.953720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:34.953803 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:34.953944 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:27:34.954267 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:34.954963 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:34.956262 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:34.956704 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:34.956810 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:34.957216 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:27:34.957871 (+ 655us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:27:34.957980 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:34.958101 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 21:27:34.958191 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 21:27:34.962610 7623 raft_consensus.cc:1261] T b2fa80d2ade2498b82237126f6255a91 P 4ff9eaae578a43f0aa15395e349d3117 [term 1 FOLLOWER]: Refusing update from remote peer 7f60711999224fc782e173cc4e4af402: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:27:34.962844 7764 raft_consensus.cc:1261] T b2fa80d2ade2498b82237126f6255a91 P 277d8b80cca4431c9917b2ea4570c077 [term 1 FOLLOWER]: Refusing update from remote peer 7f60711999224fc782e173cc4e4af402: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:27:34.963358 7924 consensus_queue.cc:1035] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ff9eaae578a43f0aa15395e349d3117" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:27:34.963927 7922 consensus_queue.cc:1035] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "277d8b80cca4431c9917b2ea4570c077" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:27:34.966779 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.966245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.2:55609)
0927 21:27:34.966479 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.966593 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260927 21:27:34.967340 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.966780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.3:43875)
0927 21:27:34.967013 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.967099 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260927 21:27:34.981537 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.981018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.1:35247)
0927 21:27:34.981232 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.981344 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260927 21:27:34.981949 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.981367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.2:55047)
0927 21:27:34.981677 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.981764 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260927 21:27:34.984129 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.983614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.3:43505)
0927 21:27:34.983829 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.983923 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 21:27:34.990962 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:34.990552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.97.1:50369)
0927 21:27:34.990746 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:34.990841 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 21:27:35.001408 7156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:27:35.001835 7156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:27:35.009816 7156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f5af1173bb24003b4cac6a9beac8b91: Sending DeleteTablet for 3 replicas of tablet b2fa80d2ade2498b82237126f6255a91
I20260927 21:27:35.011129 7744 tablet_service.cc:1514] Processing DeleteTablet for tablet b2fa80d2ade2498b82237126f6255a91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:27:35 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:58824
I20260927 21:27:35.011308 7603 tablet_service.cc:1514] Processing DeleteTablet for tablet b2fa80d2ade2498b82237126f6255a91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:27:35 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:42870
I20260927 21:27:35.015591 7458 tablet_service.cc:1514] Processing DeleteTablet for tablet b2fa80d2ade2498b82237126f6255a91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:27:35 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:46902
I20260927 21:27:35.022217 7937 tablet_replica.cc:330] T b2fa80d2ade2498b82237126f6255a91 P 277d8b80cca4431c9917b2ea4570c077: stopping tablet replica
I20260927 21:27:35.022603 7937 raft_consensus.cc:2229] T b2fa80d2ade2498b82237126f6255a91 P 277d8b80cca4431c9917b2ea4570c077 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:27:35.022997 7937 raft_consensus.cc:2258] T b2fa80d2ade2498b82237126f6255a91 P 277d8b80cca4431c9917b2ea4570c077 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:27:35.023936 7938 tablet_replica.cc:330] T b2fa80d2ade2498b82237126f6255a91 P 4ff9eaae578a43f0aa15395e349d3117: stopping tablet replica
I20260927 21:27:35.024293 7937 ts_tablet_manager.cc:1905] T b2fa80d2ade2498b82237126f6255a91 P 277d8b80cca4431c9917b2ea4570c077: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:27:35.024273 7938 raft_consensus.cc:2229] T b2fa80d2ade2498b82237126f6255a91 P 4ff9eaae578a43f0aa15395e349d3117 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:27:35.024641 7938 raft_consensus.cc:2258] T b2fa80d2ade2498b82237126f6255a91 P 4ff9eaae578a43f0aa15395e349d3117 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:27:35.026319 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 7323
I20260927 21:27:35.026561 7938 ts_tablet_manager.cc:1905] T b2fa80d2ade2498b82237126f6255a91 P 4ff9eaae578a43f0aa15395e349d3117: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:27:35.030339 7939 tablet_replica.cc:330] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402: stopping tablet replica
I20260927 21:27:35.030665 7939 raft_consensus.cc:2229] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:27:35.031394 7939 raft_consensus.cc:2258] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:27:35.032527 7939 ts_tablet_manager.cc:1905] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:27:35.032655 7937 ts_tablet_manager.cc:1918] T b2fa80d2ade2498b82237126f6255a91 P 277d8b80cca4431c9917b2ea4570c077: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:27:35.032905 7937 log.cc:1196] T b2fa80d2ade2498b82237126f6255a91 P 277d8b80cca4431c9917b2ea4570c077: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-2/wal/wals/b2fa80d2ade2498b82237126f6255a91
I20260927 21:27:35.033505 7937 ts_tablet_manager.cc:1939] T b2fa80d2ade2498b82237126f6255a91 P 277d8b80cca4431c9917b2ea4570c077: Deleting consensus metadata
I20260927 21:27:35.034946 7143 catalog_manager.cc:4841] TS 277d8b80cca4431c9917b2ea4570c077 (127.0.97.3:33731): tablet b2fa80d2ade2498b82237126f6255a91 (table test-table [id=b5f51f12b9954ddbbccd780ebbb884fb]) successfully deleted
I20260927 21:27:35.041630 7939 ts_tablet_manager.cc:1918] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:27:35.041837 7939 log.cc:1196] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-0/wal/wals/b2fa80d2ade2498b82237126f6255a91
I20260927 21:27:35.042357 7939 ts_tablet_manager.cc:1939] T b2fa80d2ade2498b82237126f6255a91 P 7f60711999224fc782e173cc4e4af402: Deleting consensus metadata
I20260927 21:27:35.043622 7143 catalog_manager.cc:4841] TS 7f60711999224fc782e173cc4e4af402 (127.0.97.1:33945): tablet b2fa80d2ade2498b82237126f6255a91 (table test-table [id=b5f51f12b9954ddbbccd780ebbb884fb]) successfully deleted
I20260927 21:27:35.044351 7938 ts_tablet_manager.cc:1918] T b2fa80d2ade2498b82237126f6255a91 P 4ff9eaae578a43f0aa15395e349d3117: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:27:35.044512 7938 log.cc:1196] T b2fa80d2ade2498b82237126f6255a91 P 4ff9eaae578a43f0aa15395e349d3117: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790544390748963-388-0/minicluster-data/ts-1/wal/wals/b2fa80d2ade2498b82237126f6255a91
I20260927 21:27:35.045001 7938 ts_tablet_manager.cc:1939] T b2fa80d2ade2498b82237126f6255a91 P 4ff9eaae578a43f0aa15395e349d3117: Deleting consensus metadata
I20260927 21:27:35.046155 7141 catalog_manager.cc:4841] TS 4ff9eaae578a43f0aa15395e349d3117 (127.0.97.2:38203): tablet b2fa80d2ade2498b82237126f6255a91 (table test-table [id=b5f51f12b9954ddbbccd780ebbb884fb]) successfully deleted
I20260927 21:27:35.056712 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:35.027393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46928 (local address 127.0.97.1:33945)
0927 21:27:35.027564 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:35.027570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:35.038185 (+ 10615us) server_negotiation.cc:407] Connection header received
0927 21:27:35.038392 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:35.038398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:35.038478 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:35.038612 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:35.040299 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:35.041145 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:35.043197 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:35.043868 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:35.043977 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:35.047720 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:35.047742 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:35.047767 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:35.047842 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:35.052744 (+ 4902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:35.053771 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:35.053792 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:35.053797 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:35.054005 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:35.054474 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:35.054480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:35.054487 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:35.055051 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:35.055395 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:35.055753 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 21:27:35.056051 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 21:27:35.057574 7518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:35.108382 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:35.107045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.0.1:33890)
0927 21:27:35.107998 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:35.108126 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":622,"threads_started":1}
I20260927 21:27:35.110052 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:35.109565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39737 (local address 127.0.0.1:33894)
0927 21:27:35.109800 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:35.109891 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 21:27:35.183732 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 7323
I20260927 21:27:35.203069 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 7535
I20260927 21:27:35.218123 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:35.203932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42882 (local address 127.0.97.2:38203)
0927 21:27:35.204142 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:35.204147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:35.204170 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:27:35.204301 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:35.204306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:35.204394 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:35.204502 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:35.205763 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:35.206518 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:35.207837 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:35.208257 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:35.208397 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:35.211422 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:35.211438 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:35.211455 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:35.211491 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:35.214887 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:35.215688 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:35.215696 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:35.215698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:35.215878 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:35.216462 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:35.216467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:35.216470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:35.216892 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:35.217192 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:35.217515 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 21:27:35.217797 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 21:27:35.218751 7663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:35.373631 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 7535
I20260927 21:27:35.387939 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 7681
I20260927 21:27:35.401448 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:35.388758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58834 (local address 127.0.97.3:33731)
0927 21:27:35.389264 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:35.389269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:35.389294 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:27:35.389344 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:35.389348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:35.389406 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:35.389524 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:35.390888 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:35.391606 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:35.392850 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:35.393257 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:35.393396 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:35.395973 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:35.395991 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:35.396007 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:35.396044 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:35.398726 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:35.399386 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:35.399393 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:35.399395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:35.399521 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:35.399908 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:35.399913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:35.399917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:35.400276 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:35.400591 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:35.400888 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 21:27:35.401102 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":229,"threads_started":1}
I20260927 21:27:35.402202 7804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:35.523833 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 7681
I20260927 21:27:35.546038 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 7019
I20260927 21:27:35.547212 7186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:35.671494 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 7019
W20260927 21:27:35.692610 7330 connection.cc:535] server connection from 127.0.0.1:53796 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 21:27:35.692955 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 7119
I20260927 21:27:35.694031 7374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:35.807794 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 7119
2026-09-27T21:27:35Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11638 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:27:35 dist-test-slave-bxm0 krb5kdc[7953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:27:35 dist-test-slave-bxm0 krb5kdc[7953](info): set up 2 sockets
Sep 27 21:27:35 dist-test-slave-bxm0 krb5kdc[7953](info): commencing operation
krb5kdc: starting...
W20260927 21:27:36.914541 388 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:27:37 dist-test-slave-bxm0 krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:27:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:27:37Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260927 21:27:37.137050 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35055
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:37.468242 7966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:37.468551 7966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:37.468710 7966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:37.476234 7966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:27:37.476367 7966 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 21:27:37.476449 7966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:37.476539 7966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:37.476629 7966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:27:37.476745 7966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:27:37.487078 7966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35055
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:37.488461 7966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:37.489871 7966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:37.506320 7971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:37.506346 7972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:37.506407 7974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:37.507571 7966 server_base.cc:1034] running on GCE node
I20260927 21:27:37.508420 7966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:37.510033 7966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:37.511255 7966 hybrid_clock.cc:648] HybridClock initialized: now 1790544457511197 us; error 41 us; skew 500 ppm
I20260927 21:27:37.511598 7966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:37 dist-test-slave-bxm0 krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544457, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:37.514832 7966 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260927 21:27:37.516402 7966 webserver.cc:460] Webserver started at http://127.0.97.62:43933/ using document root <none> and password file <none>
I20260927 21:27:37.517130 7966 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:37.517270 7966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:37.517650 7966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:37.520694 7966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "d517fba76edf42c8be94ff4072cb0616"
format_stamp: "Formatted at 2026-09-27 21:27:37 on dist-test-slave-bxm0"
server_key: "997a46035a16612df0041bc403508260"
server_key_iv: "666907f2472cf1e78f794beae3265bc8"
server_key_version: "encryptionkey@0"
I20260927 21:27:37.521584 7966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "d517fba76edf42c8be94ff4072cb0616"
format_stamp: "Formatted at 2026-09-27 21:27:37 on dist-test-slave-bxm0"
server_key: "997a46035a16612df0041bc403508260"
server_key_iv: "666907f2472cf1e78f794beae3265bc8"
server_key_version: "encryptionkey@0"
I20260927 21:27:37.527117 7966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 21:27:37.530809 7981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:37.532292 7966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:27:37.532531 7966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "d517fba76edf42c8be94ff4072cb0616"
format_stamp: "Formatted at 2026-09-27 21:27:37 on dist-test-slave-bxm0"
server_key: "997a46035a16612df0041bc403508260"
server_key_iv: "666907f2472cf1e78f794beae3265bc8"
server_key_version: "encryptionkey@0"
I20260927 21:27:37.532775 7966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:37.548152 7966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:37.552820 7966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:37.553115 7966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:37.574954 7966 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35055
I20260927 21:27:37.574980 8042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35055 every 8 connection(s)
I20260927 21:27:37.576355 7966 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:27:37.579893 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 7966
I20260927 21:27:37.580106 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:27:37.580873 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:37.580910 388 external_mini_cluster.cc:1428] Setting key b3506c29703c4b07da2e31ee297aa84a
Sep 27 21:27:37 dist-test-slave-bxm0 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 21:27:37.596727 8043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d517fba76edf42c8be94ff4072cb0616: Bootstrap starting.
I20260927 21:27:37.600571 8043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d517fba76edf42c8be94ff4072cb0616: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:37.601933 8043 log.cc:824] T 00000000000000000000000000000000 P d517fba76edf42c8be94ff4072cb0616: Log is configured to *not* fsync() on all Append() calls
I20260927 21:27:37.605135 8043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d517fba76edf42c8be94ff4072cb0616: No bootstrap required, opened a new log
I20260927 21:27:37.605599 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:37.583164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48450 (local address 127.0.97.62:35055)
0927 21:27:37.584031 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:37.584044 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:37.584087 (+ 43us) server_negotiation.cc:407] Connection header received
0927 21:27:37.585835 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:37.585860 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:37.586302 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:37.586665 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:37.588668 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:37.590080 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:37.592013 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:37.592753 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:37.592891 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:37.598093 (+ 5202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:37.598103 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:37.598119 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:37.598162 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:37.601283 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:37.602215 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:37.602222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:37.602228 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:37.602395 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:37.602792 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:37.602798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:37.602801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:37.603375 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:37.603660 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:37.604112 (+ 452us) server_negotiation.cc:299] Negotiation successful
0927 21:27:37.604527 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":426,"threads_started":1}
I20260927 21:27:37.614328 8043 raft_consensus.cc:348] T 00000000000000000000000000000000 P d517fba76edf42c8be94ff4072cb0616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d517fba76edf42c8be94ff4072cb0616" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35055 } }
I20260927 21:27:37.614737 8043 raft_consensus.cc:374] T 00000000000000000000000000000000 P d517fba76edf42c8be94ff4072cb0616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:37.614857 8043 raft_consensus.cc:729] T 00000000000000000000000000000000 P d517fba76edf42c8be94ff4072cb0616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d517fba76edf42c8be94ff4072cb0616, State: Initialized, Role: FOLLOWER
I20260927 21:27:37.615482 8043 consensus_queue.cc:260] T 00000000000000000000000000000000 P d517fba76edf42c8be94ff4072cb0616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d517fba76edf42c8be94ff4072cb0616" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35055 } }
I20260927 21:27:37.615720 8043 raft_consensus.cc:388] T 00000000000000000000000000000000 P d517fba76edf42c8be94ff4072cb0616 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:27:37.615859 8043 raft_consensus.cc:482] T 00000000000000000000000000000000 P d517fba76edf42c8be94ff4072cb0616 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:27:37.616052 8043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d517fba76edf42c8be94ff4072cb0616 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:37.618636 8043 raft_consensus.cc:504] T 00000000000000000000000000000000 P d517fba76edf42c8be94ff4072cb0616 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d517fba76edf42c8be94ff4072cb0616" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35055 } }
I20260927 21:27:37.619182 8043 leader_election.cc:302] T 00000000000000000000000000000000 P d517fba76edf42c8be94ff4072cb0616 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d517fba76edf42c8be94ff4072cb0616; no voters:
I20260927 21:27:37.619786 8043 leader_election.cc:288] T 00000000000000000000000000000000 P d517fba76edf42c8be94ff4072cb0616 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:27:37.619956 8048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d517fba76edf42c8be94ff4072cb0616 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:27:37.620864 8048 raft_consensus.cc:686] T 00000000000000000000000000000000 P d517fba76edf42c8be94ff4072cb0616 [term 1 LEADER]: Becoming Leader. State: Replica: d517fba76edf42c8be94ff4072cb0616, State: Running, Role: LEADER
I20260927 21:27:37.621719 8048 consensus_queue.cc:237] T 00000000000000000000000000000000 P d517fba76edf42c8be94ff4072cb0616 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d517fba76edf42c8be94ff4072cb0616" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35055 } }
I20260927 21:27:37.622372 8043 sys_catalog.cc:564] T 00000000000000000000000000000000 P d517fba76edf42c8be94ff4072cb0616 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:27:37.629479 8050 sys_catalog.cc:455] T 00000000000000000000000000000000 P d517fba76edf42c8be94ff4072cb0616 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d517fba76edf42c8be94ff4072cb0616. Latest consensus state: current_term: 1 leader_uuid: "d517fba76edf42c8be94ff4072cb0616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d517fba76edf42c8be94ff4072cb0616" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35055 } } }
I20260927 21:27:37.629977 8050 sys_catalog.cc:458] T 00000000000000000000000000000000 P d517fba76edf42c8be94ff4072cb0616 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:37.630571 8049 sys_catalog.cc:455] T 00000000000000000000000000000000 P d517fba76edf42c8be94ff4072cb0616 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d517fba76edf42c8be94ff4072cb0616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d517fba76edf42c8be94ff4072cb0616" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35055 } } }
I20260927 21:27:37.631017 8049 sys_catalog.cc:458] T 00000000000000000000000000000000 P d517fba76edf42c8be94ff4072cb0616 [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:37.646338 8057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:27:37.651360 8057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:27:37.661145 8057 catalog_manager.cc:1347] Generated new cluster ID: 12c2af65db944794a3d7d92362313f8e
I20260927 21:27:37.661319 8057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:27:37.679484 8057 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:27:37.680900 8057 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:27:37.690778 8057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d517fba76edf42c8be94ff4072cb0616: Generated new TSK 0
I20260927 21:27:37.691706 8057 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260927 21:27:37.757156 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35055
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:38.093947 8070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:38.094254 8070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:38.094434 8070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:38.101914 8070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:38.102032 8070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:38.102192 8070 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:27:38.102326 8070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:27:38.112422 8070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:38.113808 8070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:38.115180 8070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:38.128575 8075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:38.128602 8076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:38.128773 8078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:38.129501 8070 server_base.cc:1034] running on GCE node
I20260927 21:27:38.130187 8070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:38.131176 8070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:38.132398 8070 hybrid_clock.cc:648] HybridClock initialized: now 1790544458132312 us; error 58 us; skew 500 ppm
I20260927 21:27:38.132714 8070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:38 dist-test-slave-bxm0 krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544458, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:38.135955 8070 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 21:27:38.137589 8070 webserver.cc:460] Webserver started at http://127.0.97.1:39147/ using document root <none> and password file <none>
I20260927 21:27:38.138363 8070 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:38.138476 8070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:38.138808 8070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:38.142064 8070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-0/data/instance:
uuid: "8752312c7922404f8e197bf6305c4419"
format_stamp: "Formatted at 2026-09-27 21:27:38 on dist-test-slave-bxm0"
server_key: "a42e68ddb17a004e9f9c9432d7e44ad5"
server_key_iv: "b9a907e462d07b75b485721140656ff6"
server_key_version: "encryptionkey@0"
I20260927 21:27:38.143195 8070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8752312c7922404f8e197bf6305c4419"
format_stamp: "Formatted at 2026-09-27 21:27:38 on dist-test-slave-bxm0"
server_key: "a42e68ddb17a004e9f9c9432d7e44ad5"
server_key_iv: "b9a907e462d07b75b485721140656ff6"
server_key_version: "encryptionkey@0"
I20260927 21:27:38.149127 8070 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 21:27:38.153038 8085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:38.154553 8070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:27:38.154732 8070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "8752312c7922404f8e197bf6305c4419"
format_stamp: "Formatted at 2026-09-27 21:27:38 on dist-test-slave-bxm0"
server_key: "a42e68ddb17a004e9f9c9432d7e44ad5"
server_key_iv: "b9a907e462d07b75b485721140656ff6"
server_key_version: "encryptionkey@0"
I20260927 21:27:38.154976 8070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:38.175966 8070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:38.180790 8070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:38.181077 8070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:38.183888 8070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:38.183976 8070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:38.184115 8070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:38.184180 8070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:27:38 dist-test-slave-bxm0 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544458, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 21:27:38.211195 8094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:38.184766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33705 (local address 127.0.97.62:35055)
0927 21:27:38.185214 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:38.185221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:38.186647 (+ 1426us) server_negotiation.cc:407] Connection header received
0927 21:27:38.189964 (+ 3317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:38.189973 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:38.190078 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:38.190230 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:38.192115 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:38.193468 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:38.196143 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:38.196944 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:38.197096 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:38.202498 (+ 5402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:38.202504 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:38.202508 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:38.202543 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:38.206311 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:38.207592 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:38.207597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:38.207599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:38.207734 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:38.208368 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:38.208375 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:38.208381 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:38.208832 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:38.209032 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:38.210384 (+ 1352us) server_negotiation.cc:299] Negotiation successful
0927 21:27:38.210675 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":222,"threads_started":1}
I20260927 21:27:38.212813 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:38.185453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35055 (local address 127.0.97.1:33705)
0927 21:27:38.186437 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:38.186502 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:38.189508 (+ 3006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:38.190530 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:38.190539 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:38.191312 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:38.191919 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:38.191937 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:38.193700 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:38.193704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:38.195953 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:38.195975 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:38.197212 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:38.197218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:38.197462 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:38.198417 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:38.198445 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:38.202189 (+ 3744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:38.206579 (+ 4390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:38.206591 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:38.206619 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:38.207405 (+ 786us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:38.207866 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:38.207875 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:38.207885 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:38.208193 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:38.209128 (+ 935us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:38.209143 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:38.209795 (+ 652us) client_negotiation.cc:769] Sending connection context
0927 21:27:38.210205 (+ 410us) client_negotiation.cc:240] Negotiation successful
0927 21:27:38.211002 (+ 797us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":449,"threads_started":1}
I20260927 21:27:38.227406 8070 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39273
I20260927 21:27:38.227465 8205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39273 every 8 connection(s)
I20260927 21:27:38.228760 8070 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:27:38.231050 8206 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35055
I20260927 21:27:38.231304 8206 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:38.232154 8206 heartbeater.cc:511] Master 127.0.97.62:35055 requested a full tablet report, sending...
I20260927 21:27:38.232709 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 8070
I20260927 21:27:38.232910 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-0/wal/instance
I20260927 21:27:38.233745 388 external_mini_cluster.cc:1428] Setting key 8e0442f79b502a64b5b6be18fdce60ff
I20260927 21:27:38.234757 7998 ts_manager.cc:194] Registered new tserver with Master: 8752312c7922404f8e197bf6305c4419 (127.0.97.1:39273)
I20260927 21:27:38.235263 8190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:38.220280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35055 (local address 127.0.97.1:56531)
0927 21:27:38.221167 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:38.221195 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:38.221394 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:38.221911 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:38.221916 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:38.222711 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:38.223132 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:38.223149 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:38.224538 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:38.224546 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:38.225914 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:38.225927 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:38.226919 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:38.226926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:38.227094 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:38.227762 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:38.227782 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:38.228538 (+ 756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:38.232420 (+ 3882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:38.232429 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:38.232436 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:38.233069 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:38.233489 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:38.233494 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:38.233498 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:38.233697 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:38.234284 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:38.234289 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:38.234474 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 21:27:38.234652 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 21:27:38.234954 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"spinlock_wait_cycles":53632,"thread_start_us":618,"threads_started":1}
I20260927 21:27:38.235368 8094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:38.220834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56531 (local address 127.0.97.62:35055)
0927 21:27:38.221061 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:38.221070 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:38.221319 (+ 249us) server_negotiation.cc:407] Connection header received
0927 21:27:38.221533 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:38.221536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:38.221604 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:38.221879 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:38.223413 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:38.224330 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:38.226067 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:38.226682 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:38.226837 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:38.228737 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:38.228744 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:38.228749 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:38.228796 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:38.232290 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:38.233201 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:38.233206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:38.233210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:38.233386 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:38.233840 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:38.233853 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:38.233856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:38.234162 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:38.234328 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:38.234783 (+ 455us) server_negotiation.cc:299] Negotiation successful
0927 21:27:38.235018 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 21:27:38.237154 7998 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33705
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260927 21:27:38.286209 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35055
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:38.640676 8210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:38.641187 8210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:38.641371 8210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:38.650362 8210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:38.650486 8210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:38.650665 8210 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:27:38.650784 8210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:27:38.661427 8210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:38.662863 8210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:38.664237 8210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:38.678071 8215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:38.678038 8216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:38.678113 8218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:38.679307 8210 server_base.cc:1034] running on GCE node
I20260927 21:27:38.680030 8210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:38.681002 8210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:38.682268 8210 hybrid_clock.cc:648] HybridClock initialized: now 1790544458682186 us; error 35 us; skew 500 ppm
I20260927 21:27:38.682659 8210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:38 dist-test-slave-bxm0 krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544458, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:38.686091 8210 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 21:27:38.687678 8210 webserver.cc:460] Webserver started at http://127.0.97.2:44947/ using document root <none> and password file <none>
I20260927 21:27:38.688364 8210 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:38.688450 8210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:38.688711 8210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:38.691669 8210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-1/data/instance:
uuid: "f3c00d3e1aed43918fa13cba70ee76a6"
format_stamp: "Formatted at 2026-09-27 21:27:38 on dist-test-slave-bxm0"
server_key: "c405c87cbb579c90a9e2c388ee84f612"
server_key_iv: "f32ebf5de26d89cd20d55292f1c83b11"
server_key_version: "encryptionkey@0"
I20260927 21:27:38.692428 8210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f3c00d3e1aed43918fa13cba70ee76a6"
format_stamp: "Formatted at 2026-09-27 21:27:38 on dist-test-slave-bxm0"
server_key: "c405c87cbb579c90a9e2c388ee84f612"
server_key_iv: "f32ebf5de26d89cd20d55292f1c83b11"
server_key_version: "encryptionkey@0"
I20260927 21:27:38.697779 8210 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 21:27:38.701305 8225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:38.702842 8210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:27:38.702991 8210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "f3c00d3e1aed43918fa13cba70ee76a6"
format_stamp: "Formatted at 2026-09-27 21:27:38 on dist-test-slave-bxm0"
server_key: "c405c87cbb579c90a9e2c388ee84f612"
server_key_iv: "f32ebf5de26d89cd20d55292f1c83b11"
server_key_version: "encryptionkey@0"
I20260927 21:27:38.703177 8210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:38.728691 8210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:38.733271 8210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:38.733536 8210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:38.736506 8210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:38.736630 8210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:38.736793 8210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:38.736876 8210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:27:38 dist-test-slave-bxm0 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544458, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 21:27:38.762406 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:38.737087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34053 (local address 127.0.97.62:35055)
0927 21:27:38.737940 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:38.737946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:38.739073 (+ 1127us) server_negotiation.cc:407] Connection header received
0927 21:27:38.741289 (+ 2216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:38.741295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:38.741400 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:38.741537 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:38.744561 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:38.745575 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:38.749156 (+ 3581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:38.749591 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:38.749802 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:38.754568 (+ 4766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:38.754578 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:38.754584 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:38.754635 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:38.758116 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:38.759482 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:38.759487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:38.759490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:38.759635 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:38.760246 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:38.760253 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:38.760257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:38.760574 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:38.760719 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:38.761754 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0927 21:27:38.761987 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":508,"threads_started":1}
I20260927 21:27:38.763489 8235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:38.737910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35055 (local address 127.0.97.2:34053)
0927 21:27:38.738876 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:38.738922 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:38.740958 (+ 2036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:38.741895 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:38.741909 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:38.743072 (+ 1163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:38.744208 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:38.744253 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:38.745797 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:38.745805 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:38.748956 (+ 3151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:38.748981 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:38.750016 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:38.750024 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:38.750239 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:38.750973 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:38.750998 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:38.754238 (+ 3240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:38.758282 (+ 4044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:38.758292 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:38.758316 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:38.759326 (+ 1010us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:38.759791 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:38.759796 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:38.759800 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:38.760117 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:38.760716 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:38.760725 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:38.761289 (+ 564us) client_negotiation.cc:769] Sending connection context
0927 21:27:38.761670 (+ 381us) client_negotiation.cc:240] Negotiation successful
0927 21:27:38.762239 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":425,"threads_started":1}
I20260927 21:27:38.786783 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:38.772864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37819 (local address 127.0.97.62:35055)
0927 21:27:38.773045 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:38.773050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:38.773657 (+ 607us) server_negotiation.cc:407] Connection header received
0927 21:27:38.773903 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:38.773908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:38.773975 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:38.774084 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:38.775068 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:38.775797 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:38.777145 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:38.777551 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:38.777722 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:38.780174 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:38.780179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:38.780182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:38.780222 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:38.783526 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:38.784488 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:38.784495 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:38.784500 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:38.784667 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:38.785082 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:38.785089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:38.785093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:38.785423 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:38.785675 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:38.786369 (+ 694us) server_negotiation.cc:299] Negotiation successful
0927 21:27:38.786560 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 21:27:38.787147 8297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:38.772870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35055 (local address 127.0.97.2:37819)
0927 21:27:38.773500 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:38.773515 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:38.773720 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:38.774122 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:38.774126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:38.774627 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:38.774916 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:38.774925 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:38.775950 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:38.775956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:38.776988 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:38.776999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:38.777783 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:38.777790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:38.777987 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:38.778897 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:38.778918 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:38.779973 (+ 1055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:38.783721 (+ 3748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:38.783727 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:38.783733 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:38.784344 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:38.784817 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:38.784821 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:38.784823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:38.784961 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:38.785660 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:38.785667 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:38.786057 (+ 390us) client_negotiation.cc:769] Sending connection context
0927 21:27:38.786243 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 21:27:38.786626 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":362,"threads_started":1}
I20260927 21:27:38.800585 8210 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40457
I20260927 21:27:38.800629 8345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40457 every 8 connection(s)
I20260927 21:27:38.801892 8210 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:27:38.803974 8346 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35055
I20260927 21:27:38.804172 8346 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:38.804785 8346 heartbeater.cc:511] Master 127.0.97.62:35055 requested a full tablet report, sending...
I20260927 21:27:38.806175 7998 ts_manager.cc:194] Registered new tserver with Master: f3c00d3e1aed43918fa13cba70ee76a6 (127.0.97.2:40457)
I20260927 21:27:38.807013 7998 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34053
I20260927 21:27:38.809495 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 8210
I20260927 21:27:38.809687 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-1/wal/instance
I20260927 21:27:38.810076 388 external_mini_cluster.cc:1428] Setting key ee2fe256917db6ba83c8e9a2c4aedc38
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260927 21:27:38.857590 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35055
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:39.197443 8350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:39.197872 8350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:39.197979 8350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:39.205734 8350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:39.205871 8350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:39.206012 8350 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:27:39.206146 8350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:27:39.216852 8350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:39.218255 8350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:39.219616 8350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:39.233444 8355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:39.233470 8356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:39.233748 8358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:39.234297 8350 server_base.cc:1034] running on GCE node
I20260927 21:27:39.234918 8350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:39.235953 8350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:39.237203 8350 hybrid_clock.cc:648] HybridClock initialized: now 1790544459237132 us; error 44 us; skew 500 ppm
I20260927 21:27:39.237525 8350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:39 dist-test-slave-bxm0 krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544459, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:39.241096 8350 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 21:27:39.242823 8350 webserver.cc:460] Webserver started at http://127.0.97.3:43541/ using document root <none> and password file <none>
I20260927 21:27:39.243752 8350 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:39.243868 8350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:39.244168 8350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:39.244664 8206 heartbeater.cc:503] Master 127.0.97.62:35055 was elected leader, sending a full tablet report...
I20260927 21:27:39.248010 8350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-2/data/instance:
uuid: "38123d87872d4302a64e23c254af75e6"
format_stamp: "Formatted at 2026-09-27 21:27:39 on dist-test-slave-bxm0"
server_key: "b2aaeb76da90a0ab32fb125b5477ab77"
server_key_iv: "69709ddeabb9d9d10df2d5d6ea3c795d"
server_key_version: "encryptionkey@0"
I20260927 21:27:39.248865 8350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-2/wal/instance:
uuid: "38123d87872d4302a64e23c254af75e6"
format_stamp: "Formatted at 2026-09-27 21:27:39 on dist-test-slave-bxm0"
server_key: "b2aaeb76da90a0ab32fb125b5477ab77"
server_key_iv: "69709ddeabb9d9d10df2d5d6ea3c795d"
server_key_version: "encryptionkey@0"
I20260927 21:27:39.255316 8350 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 21:27:39.259454 8365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:39.261247 8350 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:27:39.261417 8350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "38123d87872d4302a64e23c254af75e6"
format_stamp: "Formatted at 2026-09-27 21:27:39 on dist-test-slave-bxm0"
server_key: "b2aaeb76da90a0ab32fb125b5477ab77"
server_key_iv: "69709ddeabb9d9d10df2d5d6ea3c795d"
server_key_version: "encryptionkey@0"
I20260927 21:27:39.261811 8350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:39.290644 8350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:39.295557 8350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:39.295837 8350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:39.298660 8350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:39.298749 8350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:39.298914 8350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:39.298996 8350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:27:39 dist-test-slave-bxm0 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544459, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 21:27:39.323372 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:39.299389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38551 (local address 127.0.97.62:35055)
0927 21:27:39.300005 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:39.300011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:39.301200 (+ 1189us) server_negotiation.cc:407] Connection header received
0927 21:27:39.303551 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:39.303558 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:39.303654 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:39.303738 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:39.306429 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.307609 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.309411 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.309939 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.310091 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:39.315353 (+ 5262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:39.315360 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:39.315365 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:39.315414 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:39.318980 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:39.320180 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:39.320186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:39.320192 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:39.320337 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:39.321035 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:39.321042 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:39.321048 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:39.321416 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:39.321598 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:39.322805 (+ 1207us) server_negotiation.cc:299] Negotiation successful
0927 21:27:39.322999 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"spinlock_wait_cycles":11136,"thread_start_us":433,"threads_started":1}
I20260927 21:27:39.325577 8375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:39.300102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35055 (local address 127.0.97.3:38551)
0927 21:27:39.300996 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:39.301053 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:39.303139 (+ 2086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:39.304330 (+ 1191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:39.304342 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:39.305165 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:39.306238 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:39.306260 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.307793 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.307799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:39.309246 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:39.309258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.310216 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.310223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:39.310588 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:39.311495 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:39.311546 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:39.315037 (+ 3491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:39.319156 (+ 4119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:39.319170 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:39.319198 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:39.320037 (+ 839us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:39.320479 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:39.320484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:39.320488 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:39.320729 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:39.321593 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:39.321606 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:39.322293 (+ 687us) client_negotiation.cc:769] Sending connection context
0927 21:27:39.322708 (+ 415us) client_negotiation.cc:240] Negotiation successful
0927 21:27:39.323476 (+ 768us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":366,"threads_started":1}
I20260927 21:27:39.350893 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:39.332852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35055 (local address 127.0.97.3:37993)
0927 21:27:39.333579 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:39.333602 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:39.333857 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:39.334214 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:39.334219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:39.334691 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:39.334980 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:39.334989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.336456 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.336463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:39.339040 (+ 2577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:39.339071 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.340100 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.340111 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:39.340413 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:39.341280 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:39.341320 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:39.343044 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:39.346609 (+ 3565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:39.346616 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:39.346620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:39.347327 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:39.347861 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:39.347867 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:39.347870 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:39.348192 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:39.349019 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:39.349025 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:39.349392 (+ 367us) client_negotiation.cc:769] Sending connection context
0927 21:27:39.349602 (+ 210us) client_negotiation.cc:240] Negotiation successful
0927 21:27:39.350172 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":374,"threads_started":1}
I20260927 21:27:39.351486 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:39.332823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37993 (local address 127.0.97.62:35055)
0927 21:27:39.333059 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:39.333068 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:39.333727 (+ 659us) server_negotiation.cc:407] Connection header received
0927 21:27:39.333999 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:39.334004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:39.334079 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:39.334186 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:39.335125 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.336264 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.339291 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.339831 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.340002 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:39.343358 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:39.343365 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:39.343369 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:39.343414 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:39.346412 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:39.347537 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:39.347544 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:39.347548 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:39.347722 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:39.348373 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:39.348381 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:39.348389 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:39.348890 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:39.349059 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:39.350734 (+ 1675us) server_negotiation.cc:299] Negotiation successful
0927 21:27:39.351020 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 21:27:39.356997 8350 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33407
I20260927 21:27:39.357033 8485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33407 every 8 connection(s)
I20260927 21:27:39.358356 8350 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:27:39.360741 8486 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35055
I20260927 21:27:39.361047 8486 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:39.361907 8486 heartbeater.cc:511] Master 127.0.97.62:35055 requested a full tablet report, sending...
I20260927 21:27:39.362037 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 8350
I20260927 21:27:39.362210 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-2/wal/instance
I20260927 21:27:39.362761 388 external_mini_cluster.cc:1428] Setting key 9880c15cf0ba8a8118d138717e5d815d
I20260927 21:27:39.363654 7998 ts_manager.cc:194] Registered new tserver with Master: 38123d87872d4302a64e23c254af75e6 (127.0.97.3:33407)
I20260927 21:27:39.364837 7998 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38551
I20260927 21:27:39.366011 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 21:27:39 dist-test-slave-bxm0 krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544459, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:27:39 dist-test-slave-bxm0 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:39 dist-test-slave-bxm0 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 21:27:39.416785 8374 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:39.405726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48456 (local address 127.0.97.62:35055)
0927 21:27:39.405958 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:39.405966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:39.406188 (+ 222us) server_negotiation.cc:407] Connection header received
0927 21:27:39.406403 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:39.406421 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:39.406521 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:39.406635 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:39.408072 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.408798 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.410093 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.410525 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.410657 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:39.416553 (+ 5896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48456: BlockingRecv error: recv got EOF from 127.0.0.1:48456 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 27 21:27:39 dist-test-slave-bxm0 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544459, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 21:27:39.433691 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:39.419956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48458 (local address 127.0.97.62:35055)
0927 21:27:39.420135 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:39.420140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:39.420503 (+ 363us) server_negotiation.cc:407] Connection header received
0927 21:27:39.420621 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:39.420625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:39.420694 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:39.420787 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:39.422004 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.422797 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.424121 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.424563 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.424681 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:39.428644 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:39.428650 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:39.428653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:39.428691 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:39.431259 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:39.431938 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:39.431942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:39.431944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:39.432059 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:39.432377 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:39.432380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:39.432382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:39.432653 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:39.432834 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:39.433164 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 21:27:39.433338 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 21:27:39.437711 7996 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:27:39.444916 7996 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:27:39.470939 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:39.457356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53132 (local address 127.0.97.2:40457)
0927 21:27:39.462609 (+ 5253us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:39.462623 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:39.462660 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:27:39.462779 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:39.462791 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:39.463148 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:39.463375 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:39.464206 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.465674 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.468150 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.469510 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.469710 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:39.469945 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:39.470415 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 21:27:39.470608 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5046,"thread_start_us":4893,"threads_started":1}
I20260927 21:27:39.470988 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:39.457213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40457 (local address 127.0.0.1:53132)
0927 21:27:39.461602 (+ 4389us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:39.461676 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:39.461927 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:39.463405 (+ 1478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:39.463415 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:39.463460 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:39.463981 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:39.464002 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.465833 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.465838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:39.467994 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:39.468005 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.469732 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.469738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:39.470127 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:39.470171 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 21:27:39.470351 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 21:27:39.470575 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2628,"mutex_wait_us":95,"thread_start_us":5884,"threads_started":1}
I20260927 21:27:39.472677 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:39.457322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33407 (local address 127.0.0.1:45644)
0927 21:27:39.463368 (+ 6046us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:39.463384 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:39.463584 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:39.464601 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:39.464605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:39.464678 (+ 73us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:39.465057 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:39.465068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.467247 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.467258 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:39.470143 (+ 2885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:39.470152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.471789 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.471795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:39.471965 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:39.471999 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 21:27:39.472121 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 21:27:39.472254 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5913,"mutex_wait_us":200,"thread_start_us":6457,"threads_started":1}
I20260927 21:27:39.472659 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:39.457983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45644 (local address 127.0.97.3:33407)
0927 21:27:39.459427 (+ 1444us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:39.459441 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:39.463480 (+ 4039us) server_negotiation.cc:407] Connection header received
0927 21:27:39.463817 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:39.463826 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:39.464087 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:39.464250 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:39.465328 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.467114 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.470297 (+ 3183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.471581 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.471746 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:39.471996 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:39.472165 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 21:27:39.472320 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1166,"thread_start_us":983,"threads_started":1}
I20260927 21:27:39.474687 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:39.459146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40936 (local address 127.0.97.1:39273)
0927 21:27:39.462902 (+ 3756us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:39.462915 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:39.464496 (+ 1581us) server_negotiation.cc:407] Connection header received
0927 21:27:39.464595 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:39.464601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:39.464825 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:39.465017 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:39.465820 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.467052 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.470360 (+ 3308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.472414 (+ 2054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.472573 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:39.473191 (+ 618us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:39.474217 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0927 21:27:39.474400 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3574,"thread_start_us":3449,"threads_started":1}
I20260927 21:27:39.475343 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:39.458974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39273 (local address 127.0.0.1:40936)
0927 21:27:39.464254 (+ 5280us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:39.464272 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:39.464376 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:39.465102 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:39.465106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:39.465131 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:39.465600 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:39.465637 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.467247 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.467258 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:39.470110 (+ 2852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:39.470132 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.473331 (+ 3199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.473337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:39.473523 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:39.473553 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 21:27:39.473663 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 21:27:39.474984 (+ 1321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5159,"mutex_wait_us":125,"thread_start_us":697,"threads_started":1}
I20260927 21:27:39.475582 8274 tablet_service.cc:1467] Processing CreateTablet for tablet 824405c92c0f4b3ba52daa2a5836210b (DEFAULT_TABLE table=test-table [id=47a549096de949349fba903c7533b602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:27:39.476011 8416 tablet_service.cc:1467] Processing CreateTablet for tablet 824405c92c0f4b3ba52daa2a5836210b (DEFAULT_TABLE table=test-table [id=47a549096de949349fba903c7533b602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:27:39.477109 8274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824405c92c0f4b3ba52daa2a5836210b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:39.477216 8416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824405c92c0f4b3ba52daa2a5836210b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:39.478785 8136 tablet_service.cc:1467] Processing CreateTablet for tablet 824405c92c0f4b3ba52daa2a5836210b (DEFAULT_TABLE table=test-table [id=47a549096de949349fba903c7533b602]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:27:39.480146 8136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824405c92c0f4b3ba52daa2a5836210b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:39.489007 8510 tablet_bootstrap.cc:492] T 824405c92c0f4b3ba52daa2a5836210b P 38123d87872d4302a64e23c254af75e6: Bootstrap starting.
I20260927 21:27:39.492347 8510 tablet_bootstrap.cc:654] T 824405c92c0f4b3ba52daa2a5836210b P 38123d87872d4302a64e23c254af75e6: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:39.493232 8511 tablet_bootstrap.cc:492] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6: Bootstrap starting.
I20260927 21:27:39.493800 8510 log.cc:824] T 824405c92c0f4b3ba52daa2a5836210b P 38123d87872d4302a64e23c254af75e6: Log is configured to *not* fsync() on all Append() calls
I20260927 21:27:39.495652 8513 tablet_bootstrap.cc:492] T 824405c92c0f4b3ba52daa2a5836210b P 8752312c7922404f8e197bf6305c4419: Bootstrap starting.
I20260927 21:27:39.497041 8511 tablet_bootstrap.cc:654] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:39.497195 8510 tablet_bootstrap.cc:492] T 824405c92c0f4b3ba52daa2a5836210b P 38123d87872d4302a64e23c254af75e6: No bootstrap required, opened a new log
I20260927 21:27:39.497480 8510 ts_tablet_manager.cc:1397] T 824405c92c0f4b3ba52daa2a5836210b P 38123d87872d4302a64e23c254af75e6: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 21:27:39.498487 8511 log.cc:824] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6: Log is configured to *not* fsync() on all Append() calls
I20260927 21:27:39.500152 8513 tablet_bootstrap.cc:654] T 824405c92c0f4b3ba52daa2a5836210b P 8752312c7922404f8e197bf6305c4419: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:39.501438 8513 log.cc:824] T 824405c92c0f4b3ba52daa2a5836210b P 8752312c7922404f8e197bf6305c4419: Log is configured to *not* fsync() on all Append() calls
I20260927 21:27:39.501783 8511 tablet_bootstrap.cc:492] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6: No bootstrap required, opened a new log
I20260927 21:27:39.502107 8511 ts_tablet_manager.cc:1397] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 21:27:39.504668 8513 tablet_bootstrap.cc:492] T 824405c92c0f4b3ba52daa2a5836210b P 8752312c7922404f8e197bf6305c4419: No bootstrap required, opened a new log
I20260927 21:27:39.504936 8513 ts_tablet_manager.cc:1397] T 824405c92c0f4b3ba52daa2a5836210b P 8752312c7922404f8e197bf6305c4419: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 21:27:39.505842 8510 raft_consensus.cc:348] T 824405c92c0f4b3ba52daa2a5836210b P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } } peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } }
I20260927 21:27:39.506295 8510 raft_consensus.cc:374] T 824405c92c0f4b3ba52daa2a5836210b P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:39.506422 8510 raft_consensus.cc:729] T 824405c92c0f4b3ba52daa2a5836210b P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38123d87872d4302a64e23c254af75e6, State: Initialized, Role: FOLLOWER
I20260927 21:27:39.507117 8510 consensus_queue.cc:260] T 824405c92c0f4b3ba52daa2a5836210b P 38123d87872d4302a64e23c254af75e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } } peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } }
I20260927 21:27:39.509158 8486 heartbeater.cc:503] Master 127.0.97.62:35055 was elected leader, sending a full tablet report...
I20260927 21:27:39.509917 8510 ts_tablet_manager.cc:1428] T 824405c92c0f4b3ba52daa2a5836210b P 38123d87872d4302a64e23c254af75e6: Time spent starting tablet: real 0.012s user 0.014s sys 0.000s
I20260927 21:27:39.512683 8511 raft_consensus.cc:348] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } } peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } }
I20260927 21:27:39.513130 8511 raft_consensus.cc:374] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:39.513233 8511 raft_consensus.cc:729] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c00d3e1aed43918fa13cba70ee76a6, State: Initialized, Role: FOLLOWER
I20260927 21:27:39.513898 8511 consensus_queue.cc:260] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } } peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } }
I20260927 21:27:39.515096 8346 heartbeater.cc:503] Master 127.0.97.62:35055 was elected leader, sending a full tablet report...
I20260927 21:27:39.515836 8511 ts_tablet_manager.cc:1428] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6: Time spent starting tablet: real 0.014s user 0.006s sys 0.007s
I20260927 21:27:39.515611 8513 raft_consensus.cc:348] T 824405c92c0f4b3ba52daa2a5836210b P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } } peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } }
I20260927 21:27:39.516048 8513 raft_consensus.cc:374] T 824405c92c0f4b3ba52daa2a5836210b P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:39.516156 8513 raft_consensus.cc:729] T 824405c92c0f4b3ba52daa2a5836210b P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8752312c7922404f8e197bf6305c4419, State: Initialized, Role: FOLLOWER
I20260927 21:27:39.516808 8513 consensus_queue.cc:260] T 824405c92c0f4b3ba52daa2a5836210b P 8752312c7922404f8e197bf6305c4419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } } peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } }
I20260927 21:27:39.518720 8513 ts_tablet_manager.cc:1428] T 824405c92c0f4b3ba52daa2a5836210b P 8752312c7922404f8e197bf6305c4419: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260927 21:27:39.553920 8347 tablet.cc:2369] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:27:39.610250 8487 tablet.cc:2369] T 824405c92c0f4b3ba52daa2a5836210b P 38123d87872d4302a64e23c254af75e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:27:39.664541 8517 raft_consensus.cc:482] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:27:39.665000 8517 raft_consensus.cc:504] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } } peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } }
I20260927 21:27:39.667187 8517 leader_election.cc:288] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38123d87872d4302a64e23c254af75e6 (127.0.97.3:33407), 8752312c7922404f8e197bf6305c4419 (127.0.97.1:39273)
I20260927 21:27:39.676486 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:39.667748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42029 (local address 127.0.97.1:39273)
0927 21:27:39.667920 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:39.667926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:39.668686 (+ 760us) server_negotiation.cc:407] Connection header received
0927 21:27:39.668886 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:39.668892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:39.668992 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:39.669098 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:39.670066 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.671154 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.674294 (+ 3140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.675520 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.675674 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:39.675792 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:39.676114 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 21:27:39.676250 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:27:39.676858 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:39.667703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54815 (local address 127.0.97.3:33407)
0927 21:27:39.667921 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:39.667927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:39.668685 (+ 758us) server_negotiation.cc:407] Connection header received
0927 21:27:39.668886 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:39.668892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:39.668980 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:39.669098 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:39.670060 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.671157 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.674644 (+ 3487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.675930 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.676060 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:39.676159 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:39.676474 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 21:27:39.676613 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 21:27:39.676985 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:39.667480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39273 (local address 127.0.97.2:42029)
0927 21:27:39.668573 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:39.668589 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:39.668733 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:39.669126 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:39.669131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:39.669164 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:39.669883 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:39.669895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.671311 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.671317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:39.674053 (+ 2736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:39.674073 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.675777 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.675782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:39.675917 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:39.675942 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 21:27:39.676071 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 21:27:39.676238 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":969,"mutex_wait_us":293,"spinlock_wait_cycles":22912,"thread_start_us":585,"threads_started":1}
I20260927 21:27:39.677034 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:39.667480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33407 (local address 127.0.97.2:54815)
0927 21:27:39.668510 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:39.668540 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:39.668733 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:39.669126 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:39.669131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:39.669165 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:39.669822 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:39.669850 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.671311 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.671317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:39.674052 (+ 2735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:39.674073 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.676145 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.676152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:39.676300 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:39.676330 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 21:27:39.676430 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 21:27:39.676559 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"mutex_wait_us":242,"spinlock_wait_cycles":24064,"thread_start_us":615,"threads_started":1}
I20260927 21:27:39.678138 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824405c92c0f4b3ba52daa2a5836210b" candidate_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8752312c7922404f8e197bf6305c4419" is_pre_election: true
I20260927 21:27:39.678134 8441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824405c92c0f4b3ba52daa2a5836210b" candidate_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38123d87872d4302a64e23c254af75e6" is_pre_election: true
I20260927 21:27:39.678647 8441 raft_consensus.cc:2454] T 824405c92c0f4b3ba52daa2a5836210b P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3c00d3e1aed43918fa13cba70ee76a6 in term 0.
I20260927 21:27:39.678646 8156 raft_consensus.cc:2454] T 824405c92c0f4b3ba52daa2a5836210b P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3c00d3e1aed43918fa13cba70ee76a6 in term 0.
I20260927 21:27:39.679360 8229 leader_election.cc:302] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8752312c7922404f8e197bf6305c4419, f3c00d3e1aed43918fa13cba70ee76a6; no voters:
I20260927 21:27:39.679771 8517 raft_consensus.cc:2781] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:27:39.679917 8517 raft_consensus.cc:482] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:27:39.680029 8517 raft_consensus.cc:3037] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:39.682282 8517 raft_consensus.cc:504] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } } peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } }
I20260927 21:27:39.682976 8517 leader_election.cc:288] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6 [CANDIDATE]: Term 1 election: Requested vote from peers 38123d87872d4302a64e23c254af75e6 (127.0.97.3:33407), 8752312c7922404f8e197bf6305c4419 (127.0.97.1:39273)
I20260927 21:27:39.683568 8441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824405c92c0f4b3ba52daa2a5836210b" candidate_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38123d87872d4302a64e23c254af75e6"
I20260927 21:27:39.683645 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824405c92c0f4b3ba52daa2a5836210b" candidate_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8752312c7922404f8e197bf6305c4419"
I20260927 21:27:39.683866 8156 raft_consensus.cc:3037] T 824405c92c0f4b3ba52daa2a5836210b P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:39.683866 8441 raft_consensus.cc:3037] T 824405c92c0f4b3ba52daa2a5836210b P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:39.687316 8441 raft_consensus.cc:2454] T 824405c92c0f4b3ba52daa2a5836210b P 38123d87872d4302a64e23c254af75e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c00d3e1aed43918fa13cba70ee76a6 in term 1.
I20260927 21:27:39.687316 8156 raft_consensus.cc:2454] T 824405c92c0f4b3ba52daa2a5836210b P 8752312c7922404f8e197bf6305c4419 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c00d3e1aed43918fa13cba70ee76a6 in term 1.
I20260927 21:27:39.687889 8229 leader_election.cc:302] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 38123d87872d4302a64e23c254af75e6, 8752312c7922404f8e197bf6305c4419, f3c00d3e1aed43918fa13cba70ee76a6; no voters:
I20260927 21:27:39.688262 8517 raft_consensus.cc:2781] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:27:39.688720 8517 raft_consensus.cc:686] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6 [term 1 LEADER]: Becoming Leader. State: Replica: f3c00d3e1aed43918fa13cba70ee76a6, State: Running, Role: LEADER
I20260927 21:27:39.689312 8517 consensus_queue.cc:237] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } } peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } }
I20260927 21:27:39.693813 7998 catalog_manager.cc:5495] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3c00d3e1aed43918fa13cba70ee76a6 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } health_report { overall_health: UNKNOWN } } }
W20260927 21:27:39.731987 8207 tablet.cc:2369] T 824405c92c0f4b3ba52daa2a5836210b P 8752312c7922404f8e197bf6305c4419: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:27:39.764199 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:39.758952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53140 (local address 127.0.97.2:40457)
0927 21:27:39.759134 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:39.759142 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:39.759184 (+ 42us) server_negotiation.cc:407] Connection header received
0927 21:27:39.759465 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:39.759469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:39.759543 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:39.759716 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:27:39.760121 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.760831 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.762203 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:39.762605 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:39.762710 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:39.763154 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:27:39.763692 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:27:39.763809 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:39.763916 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 21:27:39.763983 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 21:27:39.772361 8156 raft_consensus.cc:1261] T 824405c92c0f4b3ba52daa2a5836210b P 8752312c7922404f8e197bf6305c4419 [term 1 FOLLOWER]: Refusing update from remote peer f3c00d3e1aed43918fa13cba70ee76a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:27:39.772444 8441 raft_consensus.cc:1261] T 824405c92c0f4b3ba52daa2a5836210b P 38123d87872d4302a64e23c254af75e6 [term 1 FOLLOWER]: Refusing update from remote peer f3c00d3e1aed43918fa13cba70ee76a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:27:39.773453 8521 consensus_queue.cc:1035] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:27:39.773873 8517 consensus_queue.cc:1035] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:27:39.798931 7996 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:27:39.799238 7996 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:27:39.803412 7996 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d517fba76edf42c8be94ff4072cb0616: Sending DeleteTablet for 3 replicas of tablet 824405c92c0f4b3ba52daa2a5836210b
I20260927 21:27:39.804613 8274 tablet_service.cc:1514] Processing DeleteTablet for tablet 824405c92c0f4b3ba52daa2a5836210b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:27:39 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53132
I20260927 21:27:39.804613 8416 tablet_service.cc:1514] Processing DeleteTablet for tablet 824405c92c0f4b3ba52daa2a5836210b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:27:39 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45644
I20260927 21:27:39.804927 8136 tablet_service.cc:1514] Processing DeleteTablet for tablet 824405c92c0f4b3ba52daa2a5836210b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:27:39 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40936
I20260927 21:27:39.806048 7996 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:27:39.806910 7996 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:27:39.823006 8536 tablet_replica.cc:330] T 824405c92c0f4b3ba52daa2a5836210b P 8752312c7922404f8e197bf6305c4419: stopping tablet replica
I20260927 21:27:39.823415 8135 tablet_service.cc:1467] Processing CreateTablet for tablet 31a917485f2140dab76ac82be7529969 (DEFAULT_TABLE table=test-table [id=337e476d3eb2415b9385553b3be688a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:27:39.823676 8536 raft_consensus.cc:2229] T 824405c92c0f4b3ba52daa2a5836210b P 8752312c7922404f8e197bf6305c4419 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:27:39.823658 8415 tablet_service.cc:1467] Processing CreateTablet for tablet 31a917485f2140dab76ac82be7529969 (DEFAULT_TABLE table=test-table [id=337e476d3eb2415b9385553b3be688a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:27:39.824156 8135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31a917485f2140dab76ac82be7529969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:39.824246 8415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31a917485f2140dab76ac82be7529969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:39.824239 8536 raft_consensus.cc:2258] T 824405c92c0f4b3ba52daa2a5836210b P 8752312c7922404f8e197bf6305c4419 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:27:39.827219 8276 tablet_service.cc:1467] Processing CreateTablet for tablet 31a917485f2140dab76ac82be7529969 (DEFAULT_TABLE table=test-table [id=337e476d3eb2415b9385553b3be688a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:27:39.827817 8276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31a917485f2140dab76ac82be7529969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:39.830922 8535 tablet_replica.cc:330] T 824405c92c0f4b3ba52daa2a5836210b P 38123d87872d4302a64e23c254af75e6: stopping tablet replica
I20260927 21:27:39.831436 8535 raft_consensus.cc:2229] T 824405c92c0f4b3ba52daa2a5836210b P 38123d87872d4302a64e23c254af75e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:27:39.831980 8535 raft_consensus.cc:2258] T 824405c92c0f4b3ba52daa2a5836210b P 38123d87872d4302a64e23c254af75e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:27:39.832358 8536 ts_tablet_manager.cc:1905] T 824405c92c0f4b3ba52daa2a5836210b P 8752312c7922404f8e197bf6305c4419: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:27:39.834240 8535 ts_tablet_manager.cc:1905] T 824405c92c0f4b3ba52daa2a5836210b P 38123d87872d4302a64e23c254af75e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:27:39.834689 8513 tablet_bootstrap.cc:492] T 31a917485f2140dab76ac82be7529969 P 8752312c7922404f8e197bf6305c4419: Bootstrap starting.
I20260927 21:27:39.834735 8510 tablet_bootstrap.cc:492] T 31a917485f2140dab76ac82be7529969 P 38123d87872d4302a64e23c254af75e6: Bootstrap starting.
I20260927 21:27:39.836431 8513 tablet_bootstrap.cc:654] T 31a917485f2140dab76ac82be7529969 P 8752312c7922404f8e197bf6305c4419: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:39.836513 8510 tablet_bootstrap.cc:654] T 31a917485f2140dab76ac82be7529969 P 38123d87872d4302a64e23c254af75e6: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:39.843538 8534 tablet_replica.cc:330] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6: stopping tablet replica
I20260927 21:27:39.844184 8534 raft_consensus.cc:2229] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:27:39.845324 8534 raft_consensus.cc:2258] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:27:39.846894 8535 ts_tablet_manager.cc:1918] T 824405c92c0f4b3ba52daa2a5836210b P 38123d87872d4302a64e23c254af75e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:27:39.847065 8535 log.cc:1196] T 824405c92c0f4b3ba52daa2a5836210b P 38123d87872d4302a64e23c254af75e6: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-2/wal/wals/824405c92c0f4b3ba52daa2a5836210b
I20260927 21:27:39.847569 8535 ts_tablet_manager.cc:1939] T 824405c92c0f4b3ba52daa2a5836210b P 38123d87872d4302a64e23c254af75e6: Deleting consensus metadata
I20260927 21:27:39.849475 7982 catalog_manager.cc:4841] TS 38123d87872d4302a64e23c254af75e6 (127.0.97.3:33407): tablet 824405c92c0f4b3ba52daa2a5836210b (table test-table [id=47a549096de949349fba903c7533b602]) successfully deleted
I20260927 21:27:39.850063 8534 ts_tablet_manager.cc:1905] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:27:39.850978 8510 tablet_bootstrap.cc:492] T 31a917485f2140dab76ac82be7529969 P 38123d87872d4302a64e23c254af75e6: No bootstrap required, opened a new log
I20260927 21:27:39.851136 8510 ts_tablet_manager.cc:1397] T 31a917485f2140dab76ac82be7529969 P 38123d87872d4302a64e23c254af75e6: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.004s
I20260927 21:27:39.852069 8510 raft_consensus.cc:348] T 31a917485f2140dab76ac82be7529969 P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:39.852296 8510 raft_consensus.cc:374] T 31a917485f2140dab76ac82be7529969 P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:39.852399 8510 raft_consensus.cc:729] T 31a917485f2140dab76ac82be7529969 P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38123d87872d4302a64e23c254af75e6, State: Initialized, Role: FOLLOWER
I20260927 21:27:39.852679 8510 consensus_queue.cc:260] T 31a917485f2140dab76ac82be7529969 P 38123d87872d4302a64e23c254af75e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:39.852907 8511 tablet_bootstrap.cc:492] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6: Bootstrap starting.
I20260927 21:27:39.854748 8511 tablet_bootstrap.cc:654] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:39.855587 8510 ts_tablet_manager.cc:1428] T 31a917485f2140dab76ac82be7529969 P 38123d87872d4302a64e23c254af75e6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 21:27:39.860671 8513 tablet_bootstrap.cc:492] T 31a917485f2140dab76ac82be7529969 P 8752312c7922404f8e197bf6305c4419: No bootstrap required, opened a new log
I20260927 21:27:39.861393 8513 ts_tablet_manager.cc:1397] T 31a917485f2140dab76ac82be7529969 P 8752312c7922404f8e197bf6305c4419: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.008s
I20260927 21:27:39.863672 8536 ts_tablet_manager.cc:1918] T 824405c92c0f4b3ba52daa2a5836210b P 8752312c7922404f8e197bf6305c4419: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:27:39.864635 8513 raft_consensus.cc:348] T 31a917485f2140dab76ac82be7529969 P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:39.865805 8513 raft_consensus.cc:374] T 31a917485f2140dab76ac82be7529969 P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:39.865192 8536 log.cc:1196] T 824405c92c0f4b3ba52daa2a5836210b P 8752312c7922404f8e197bf6305c4419: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-0/wal/wals/824405c92c0f4b3ba52daa2a5836210b
I20260927 21:27:39.866767 8511 tablet_bootstrap.cc:492] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6: No bootstrap required, opened a new log
I20260927 21:27:39.865962 8513 raft_consensus.cc:729] T 31a917485f2140dab76ac82be7529969 P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8752312c7922404f8e197bf6305c4419, State: Initialized, Role: FOLLOWER
I20260927 21:27:39.866937 8511 ts_tablet_manager.cc:1397] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.008s
I20260927 21:27:39.867182 8513 consensus_queue.cc:260] T 31a917485f2140dab76ac82be7529969 P 8752312c7922404f8e197bf6305c4419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:39.867619 8534 ts_tablet_manager.cc:1918] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:27:39.867681 8536 ts_tablet_manager.cc:1939] T 824405c92c0f4b3ba52daa2a5836210b P 8752312c7922404f8e197bf6305c4419: Deleting consensus metadata
I20260927 21:27:39.867846 8513 ts_tablet_manager.cc:1428] T 31a917485f2140dab76ac82be7529969 P 8752312c7922404f8e197bf6305c4419: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260927 21:27:39.868512 8511 raft_consensus.cc:348] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:39.868738 8511 raft_consensus.cc:374] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:39.868916 8511 raft_consensus.cc:729] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c00d3e1aed43918fa13cba70ee76a6, State: Initialized, Role: FOLLOWER
I20260927 21:27:39.869158 8511 consensus_queue.cc:260] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:39.869689 7985 catalog_manager.cc:4841] TS 8752312c7922404f8e197bf6305c4419 (127.0.97.1:39273): tablet 824405c92c0f4b3ba52daa2a5836210b (table test-table [id=47a549096de949349fba903c7533b602]) successfully deleted
I20260927 21:27:39.870316 8511 ts_tablet_manager.cc:1428] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 21:27:39.870536 8534 log.cc:1196] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-1/wal/wals/824405c92c0f4b3ba52daa2a5836210b
I20260927 21:27:39.871008 8534 ts_tablet_manager.cc:1939] T 824405c92c0f4b3ba52daa2a5836210b P f3c00d3e1aed43918fa13cba70ee76a6: Deleting consensus metadata
I20260927 21:27:39.872292 7984 catalog_manager.cc:4841] TS f3c00d3e1aed43918fa13cba70ee76a6 (127.0.97.2:40457): tablet 824405c92c0f4b3ba52daa2a5836210b (table test-table [id=47a549096de949349fba903c7533b602]) successfully deleted
I20260927 21:27:40.014144 8529 raft_consensus.cc:482] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:27:40.014422 8529 raft_consensus.cc:504] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:40.015398 8529 leader_election.cc:288] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8752312c7922404f8e197bf6305c4419 (127.0.97.1:39273), 38123d87872d4302a64e23c254af75e6 (127.0.97.3:33407)
I20260927 21:27:40.016018 8441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a917485f2140dab76ac82be7529969" candidate_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38123d87872d4302a64e23c254af75e6" is_pre_election: true
I20260927 21:27:40.016021 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a917485f2140dab76ac82be7529969" candidate_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8752312c7922404f8e197bf6305c4419" is_pre_election: true
I20260927 21:27:40.016294 8441 raft_consensus.cc:2454] T 31a917485f2140dab76ac82be7529969 P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3c00d3e1aed43918fa13cba70ee76a6 in term 0.
I20260927 21:27:40.016306 8156 raft_consensus.cc:2454] T 31a917485f2140dab76ac82be7529969 P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3c00d3e1aed43918fa13cba70ee76a6 in term 0.
I20260927 21:27:40.016811 8229 leader_election.cc:302] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8752312c7922404f8e197bf6305c4419, f3c00d3e1aed43918fa13cba70ee76a6; no voters:
I20260927 21:27:40.017133 8529 raft_consensus.cc:2781] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:27:40.017319 8529 raft_consensus.cc:482] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:27:40.017419 8529 raft_consensus.cc:3037] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:40.019846 8529 raft_consensus.cc:504] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:40.020548 8529 leader_election.cc:288] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6 [CANDIDATE]: Term 1 election: Requested vote from peers 8752312c7922404f8e197bf6305c4419 (127.0.97.1:39273), 38123d87872d4302a64e23c254af75e6 (127.0.97.3:33407)
I20260927 21:27:40.021082 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a917485f2140dab76ac82be7529969" candidate_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8752312c7922404f8e197bf6305c4419"
I20260927 21:27:40.021143 8441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a917485f2140dab76ac82be7529969" candidate_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38123d87872d4302a64e23c254af75e6"
I20260927 21:27:40.021312 8156 raft_consensus.cc:3037] T 31a917485f2140dab76ac82be7529969 P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:40.021368 8441 raft_consensus.cc:3037] T 31a917485f2140dab76ac82be7529969 P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:40.023773 8156 raft_consensus.cc:2454] T 31a917485f2140dab76ac82be7529969 P 8752312c7922404f8e197bf6305c4419 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c00d3e1aed43918fa13cba70ee76a6 in term 1.
I20260927 21:27:40.024354 8441 raft_consensus.cc:2454] T 31a917485f2140dab76ac82be7529969 P 38123d87872d4302a64e23c254af75e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c00d3e1aed43918fa13cba70ee76a6 in term 1.
I20260927 21:27:40.024363 8229 leader_election.cc:302] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8752312c7922404f8e197bf6305c4419, f3c00d3e1aed43918fa13cba70ee76a6; no voters:
I20260927 21:27:40.024737 8529 raft_consensus.cc:2781] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:27:40.024915 8529 raft_consensus.cc:686] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6 [term 1 LEADER]: Becoming Leader. State: Replica: f3c00d3e1aed43918fa13cba70ee76a6, State: Running, Role: LEADER
I20260927 21:27:40.025231 8529 consensus_queue.cc:237] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:40.028303 7998 catalog_manager.cc:5495] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3c00d3e1aed43918fa13cba70ee76a6 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } health_report { overall_health: UNKNOWN } } }
Sep 27 21:27:40 dist-test-slave-bxm0 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544457, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 21:27:40.114476 8546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:40.100293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35055 (local address 127.0.0.1:48470)
0927 21:27:40.101044 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:40.101069 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:40.101248 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:40.101679 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:40.101686 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:40.102337 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:40.102511 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:40.102521 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.103568 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.103574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:40.104509 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:40.104519 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.105211 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.105217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:40.105390 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:40.106165 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:40.106183 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:40.108758 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:40.111896 (+ 3138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:40.111909 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:40.111918 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:40.112342 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:40.112711 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:40.112717 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:40.112721 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:40.112861 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:40.113366 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:40.113374 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:40.113540 (+ 166us) client_negotiation.cc:769] Sending connection context
0927 21:27:40.113733 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 21:27:40.114034 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":385,"threads_started":1}
I20260927 21:27:40.114504 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:40.100375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48470 (local address 127.0.97.62:35055)
0927 21:27:40.101353 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:40.101360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:40.101385 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:27:40.101436 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:40.101440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:40.101490 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:40.101603 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:40.102656 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.103421 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.104652 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.105051 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.105191 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:40.108957 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:40.108962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:40.108966 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:40.108993 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:40.111710 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:40.112463 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:40.112470 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:40.112474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:40.112606 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:40.112970 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:40.112975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:40.112978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:40.113236 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:40.113388 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:40.113829 (+ 441us) server_negotiation.cc:299] Negotiation successful
0927 21:27:40.114164 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":144,"server-negotiator.queue_time_us":821,"thread_start_us":669,"threads_started":1}
I20260927 21:27:40.120195 7998 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48470:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 21:27:40.121505 7998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:27:40.129243 8135 tablet_service.cc:1467] Processing CreateTablet for tablet ff085229dab2450680ba9e7028a3499d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f047673a21de49a48d6dce0629c910a1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:27:40.129590 8276 tablet_service.cc:1467] Processing CreateTablet for tablet ff085229dab2450680ba9e7028a3499d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f047673a21de49a48d6dce0629c910a1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:27:40.129859 8135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff085229dab2450680ba9e7028a3499d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:40.129863 8415 tablet_service.cc:1467] Processing CreateTablet for tablet ff085229dab2450680ba9e7028a3499d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f047673a21de49a48d6dce0629c910a1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:27:40.130091 8276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff085229dab2450680ba9e7028a3499d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:40.130357 8415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff085229dab2450680ba9e7028a3499d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:40.137693 8513 tablet_bootstrap.cc:492] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419: Bootstrap starting.
I20260927 21:27:40.138445 8511 tablet_bootstrap.cc:492] T ff085229dab2450680ba9e7028a3499d P f3c00d3e1aed43918fa13cba70ee76a6: Bootstrap starting.
I20260927 21:27:40.139583 8513 tablet_bootstrap.cc:654] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:40.139876 8511 tablet_bootstrap.cc:654] T ff085229dab2450680ba9e7028a3499d P f3c00d3e1aed43918fa13cba70ee76a6: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:40.141887 8513 tablet_bootstrap.cc:492] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419: No bootstrap required, opened a new log
I20260927 21:27:40.142288 8510 tablet_bootstrap.cc:492] T ff085229dab2450680ba9e7028a3499d P 38123d87872d4302a64e23c254af75e6: Bootstrap starting.
I20260927 21:27:40.144012 8510 tablet_bootstrap.cc:654] T ff085229dab2450680ba9e7028a3499d P 38123d87872d4302a64e23c254af75e6: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:40.144170 8511 tablet_bootstrap.cc:492] T ff085229dab2450680ba9e7028a3499d P f3c00d3e1aed43918fa13cba70ee76a6: No bootstrap required, opened a new log
I20260927 21:27:40.144377 8511 ts_tablet_manager.cc:1397] T ff085229dab2450680ba9e7028a3499d P f3c00d3e1aed43918fa13cba70ee76a6: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 21:27:40.142063 8513 ts_tablet_manager.cc:1397] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260927 21:27:40.145172 8511 raft_consensus.cc:348] T ff085229dab2450680ba9e7028a3499d P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:40.145745 8511 raft_consensus.cc:374] T ff085229dab2450680ba9e7028a3499d P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:40.145869 8511 raft_consensus.cc:729] T ff085229dab2450680ba9e7028a3499d P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c00d3e1aed43918fa13cba70ee76a6, State: Initialized, Role: FOLLOWER
I20260927 21:27:40.146066 8513 raft_consensus.cc:348] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:40.146136 8511 consensus_queue.cc:260] T ff085229dab2450680ba9e7028a3499d P f3c00d3e1aed43918fa13cba70ee76a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:40.146312 8513 raft_consensus.cc:374] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:40.146399 8513 raft_consensus.cc:729] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8752312c7922404f8e197bf6305c4419, State: Initialized, Role: FOLLOWER
I20260927 21:27:40.146555 8529 tablet_replica.cc:439] T ff085229dab2450680ba9e7028a3499d P f3c00d3e1aed43918fa13cba70ee76a6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } } }
I20260927 21:27:40.146831 8529 tablet_replica.cc:442] T ff085229dab2450680ba9e7028a3499d P f3c00d3e1aed43918fa13cba70ee76a6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:27:40.146704 8513 consensus_queue.cc:260] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:40.147312 8511 ts_tablet_manager.cc:1428] T ff085229dab2450680ba9e7028a3499d P f3c00d3e1aed43918fa13cba70ee76a6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 21:27:40.147228 8518 tablet_replica.cc:439] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } } }
I20260927 21:27:40.147588 8518 tablet_replica.cc:442] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:27:40.148676 8513 ts_tablet_manager.cc:1428] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 21:27:40.150004 8510 tablet_bootstrap.cc:492] T ff085229dab2450680ba9e7028a3499d P 38123d87872d4302a64e23c254af75e6: No bootstrap required, opened a new log
I20260927 21:27:40.150184 8510 ts_tablet_manager.cc:1397] T ff085229dab2450680ba9e7028a3499d P 38123d87872d4302a64e23c254af75e6: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260927 21:27:40.150947 8510 raft_consensus.cc:348] T ff085229dab2450680ba9e7028a3499d P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:40.151199 8510 raft_consensus.cc:374] T ff085229dab2450680ba9e7028a3499d P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:40.151280 8510 raft_consensus.cc:729] T ff085229dab2450680ba9e7028a3499d P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38123d87872d4302a64e23c254af75e6, State: Initialized, Role: FOLLOWER
I20260927 21:27:40.151517 8510 consensus_queue.cc:260] T ff085229dab2450680ba9e7028a3499d P 38123d87872d4302a64e23c254af75e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:40.152907 8516 tablet_replica.cc:439] T ff085229dab2450680ba9e7028a3499d P 38123d87872d4302a64e23c254af75e6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } } }
I20260927 21:27:40.153229 8516 tablet_replica.cc:442] T ff085229dab2450680ba9e7028a3499d P 38123d87872d4302a64e23c254af75e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:27:40.154192 8510 ts_tablet_manager.cc:1428] T ff085229dab2450680ba9e7028a3499d P 38123d87872d4302a64e23c254af75e6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 21:27:40.167308 8518 raft_consensus.cc:482] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:27:40.167555 8518 raft_consensus.cc:504] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:40.169759 8518 leader_election.cc:288] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3c00d3e1aed43918fa13cba70ee76a6 (127.0.97.2:40457), 38123d87872d4302a64e23c254af75e6 (127.0.97.3:33407)
I20260927 21:27:40.178632 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:40.170345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55581 (local address 127.0.97.3:33407)
0927 21:27:40.170582 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:40.170592 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:40.170675 (+ 83us) server_negotiation.cc:407] Connection header received
0927 21:27:40.170994 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:40.171001 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:40.171087 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:40.171157 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:40.172075 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.173229 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.176034 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.177263 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.177427 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:40.177579 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:40.178150 (+ 571us) server_negotiation.cc:299] Negotiation successful
0927 21:27:40.178306 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 21:27:40.178699 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:40.170882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42539 (local address 127.0.97.2:40457)
0927 21:27:40.171119 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:40.171129 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:40.172031 (+ 902us) server_negotiation.cc:407] Connection header received
0927 21:27:40.172327 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:40.172333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:40.172408 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:40.172510 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:40.173383 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.174308 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.176233 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.177584 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.177982 (+ 398us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:40.178092 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:40.178222 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 21:27:40.178408 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 21:27:40.179039 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:40.169986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40457 (local address 127.0.97.1:42539)
0927 21:27:40.171933 (+ 1947us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:40.171952 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:40.172134 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:40.172721 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:40.172725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:40.172756 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:40.173211 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:40.173221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.174461 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.174470 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:40.176039 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:40.176053 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.177823 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.177829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:40.178065 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:40.178115 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 21:27:40.178200 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 21:27:40.178349 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1793,"mutex_wait_us":2,"thread_start_us":2831,"threads_started":1}
I20260927 21:27:40.180294 8551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:40.169911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33407 (local address 127.0.97.1:55581)
0927 21:27:40.170550 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:40.170573 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:40.170795 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:40.171387 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:40.171392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:40.171420 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:40.171851 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:40.171867 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.173408 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.173415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:40.175837 (+ 2422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:40.175863 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.177712 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.177720 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:40.177935 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:40.177979 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 21:27:40.179539 (+ 1560us) client_negotiation.cc:240] Negotiation successful
0927 21:27:40.179703 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":27,"thread_start_us":1218,"threads_started":1}
I20260927 21:27:40.180140 8301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff085229dab2450680ba9e7028a3499d" candidate_uuid: "8752312c7922404f8e197bf6305c4419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" is_pre_election: true
I20260927 21:27:40.180553 8301 raft_consensus.cc:2454] T ff085229dab2450680ba9e7028a3499d P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8752312c7922404f8e197bf6305c4419 in term 0.
I20260927 21:27:40.180961 8441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff085229dab2450680ba9e7028a3499d" candidate_uuid: "8752312c7922404f8e197bf6305c4419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38123d87872d4302a64e23c254af75e6" is_pre_election: true
I20260927 21:27:40.181239 8441 raft_consensus.cc:2454] T ff085229dab2450680ba9e7028a3499d P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8752312c7922404f8e197bf6305c4419 in term 0.
I20260927 21:27:40.181201 8088 leader_election.cc:302] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8752312c7922404f8e197bf6305c4419, f3c00d3e1aed43918fa13cba70ee76a6; no voters:
I20260927 21:27:40.181651 8518 raft_consensus.cc:2781] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:27:40.181804 8518 raft_consensus.cc:482] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:27:40.181921 8518 raft_consensus.cc:3037] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:40.184180 8518 raft_consensus.cc:504] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:40.184964 8518 leader_election.cc:288] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419 [CANDIDATE]: Term 1 election: Requested vote from peers f3c00d3e1aed43918fa13cba70ee76a6 (127.0.97.2:40457), 38123d87872d4302a64e23c254af75e6 (127.0.97.3:33407)
I20260927 21:27:40.185482 8441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff085229dab2450680ba9e7028a3499d" candidate_uuid: "8752312c7922404f8e197bf6305c4419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38123d87872d4302a64e23c254af75e6"
I20260927 21:27:40.185537 8301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff085229dab2450680ba9e7028a3499d" candidate_uuid: "8752312c7922404f8e197bf6305c4419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c00d3e1aed43918fa13cba70ee76a6"
I20260927 21:27:40.185811 8441 raft_consensus.cc:3037] T ff085229dab2450680ba9e7028a3499d P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:40.185813 8301 raft_consensus.cc:3037] T ff085229dab2450680ba9e7028a3499d P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:40.189071 8441 raft_consensus.cc:2454] T ff085229dab2450680ba9e7028a3499d P 38123d87872d4302a64e23c254af75e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8752312c7922404f8e197bf6305c4419 in term 1.
I20260927 21:27:40.189589 8301 raft_consensus.cc:2454] T ff085229dab2450680ba9e7028a3499d P f3c00d3e1aed43918fa13cba70ee76a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8752312c7922404f8e197bf6305c4419 in term 1.
I20260927 21:27:40.189688 8086 leader_election.cc:302] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38123d87872d4302a64e23c254af75e6, 8752312c7922404f8e197bf6305c4419; no voters:
I20260927 21:27:40.190068 8518 raft_consensus.cc:2781] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:27:40.190559 8518 raft_consensus.cc:686] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419 [term 1 LEADER]: Becoming Leader. State: Replica: 8752312c7922404f8e197bf6305c4419, State: Running, Role: LEADER
I20260927 21:27:40.191016 8518 consensus_queue.cc:237] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:40.193593 8553 tablet_replica.cc:439] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419: TxnStatusTablet state changed. Reason: New leader 8752312c7922404f8e197bf6305c4419. Latest consensus state: current_term: 1 leader_uuid: "8752312c7922404f8e197bf6305c4419" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } } }
I20260927 21:27:40.193976 8553 tablet_replica.cc:442] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:27:40.194713 8555 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 21:27:40.194950 8555 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 21:27:40.195289 7997 catalog_manager.cc:5495] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8752312c7922404f8e197bf6305c4419 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "8752312c7922404f8e197bf6305c4419" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } health_report { overall_health: UNKNOWN } } }
I20260927 21:27:40.220884 8546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:40.209251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39273 (local address 127.0.0.1:40948)
0927 21:27:40.209784 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:40.209814 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:40.210014 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:40.214655 (+ 4641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:40.214660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:40.214678 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:27:40.214923 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:40.214932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.216420 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.216427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:40.217923 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:40.217940 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.218914 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.218921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:40.219110 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:40.219176 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:27:40.220216 (+ 1040us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:27:40.220241 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 21:27:40.220367 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 21:27:40.220540 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276}
I20260927 21:27:40.220942 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:40.209233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40948 (local address 127.0.97.1:39273)
0927 21:27:40.214237 (+ 5004us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:40.214242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:40.214273 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:27:40.214360 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:40.214364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:40.214444 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:40.214630 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:27:40.215159 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.216274 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.218111 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.218717 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.218878 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:40.219443 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:27:40.220091 (+ 648us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:27:40.220258 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:40.220415 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 21:27:40.220502 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4874,"thread_start_us":4693,"threads_started":1}
I20260927 21:27:40.225672 8441 raft_consensus.cc:1261] T ff085229dab2450680ba9e7028a3499d P 38123d87872d4302a64e23c254af75e6 [term 1 FOLLOWER]: Refusing update from remote peer 8752312c7922404f8e197bf6305c4419: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:27:40.226016 8301 raft_consensus.cc:1261] T ff085229dab2450680ba9e7028a3499d P f3c00d3e1aed43918fa13cba70ee76a6 [term 1 FOLLOWER]: Refusing update from remote peer 8752312c7922404f8e197bf6305c4419: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:27:40.226588 8553 consensus_queue.cc:1035] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:27:40.227051 8518 consensus_queue.cc:1035] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:27:40.238178 8516 tablet_replica.cc:439] T ff085229dab2450680ba9e7028a3499d P 38123d87872d4302a64e23c254af75e6: TxnStatusTablet state changed. Reason: New leader 8752312c7922404f8e197bf6305c4419. Latest consensus state: current_term: 1 leader_uuid: "8752312c7922404f8e197bf6305c4419" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } } }
I20260927 21:27:40.238560 8516 tablet_replica.cc:442] T ff085229dab2450680ba9e7028a3499d P 38123d87872d4302a64e23c254af75e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:27:40.244117 8553 tablet_replica.cc:439] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8752312c7922404f8e197bf6305c4419" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } } }
I20260927 21:27:40.246937 8525 mvcc.cc:205] Tried to move back new op lower bound from 7334070109076926464 to 7334070108946796544. Current Snapshot: MvccSnapshot[applied={T|T < 7334070109076926464 or (T in {7334070109076926464})}]
I20260927 21:27:40.247679 8516 tablet_replica.cc:439] T ff085229dab2450680ba9e7028a3499d P 38123d87872d4302a64e23c254af75e6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8752312c7922404f8e197bf6305c4419" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } } }
I20260927 21:27:40.248028 8516 tablet_replica.cc:442] T ff085229dab2450680ba9e7028a3499d P 38123d87872d4302a64e23c254af75e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:27:40.248123 8553 tablet_replica.cc:442] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:27:40.262441 8529 tablet_replica.cc:439] T ff085229dab2450680ba9e7028a3499d P f3c00d3e1aed43918fa13cba70ee76a6: TxnStatusTablet state changed. Reason: New leader 8752312c7922404f8e197bf6305c4419. Latest consensus state: current_term: 1 leader_uuid: "8752312c7922404f8e197bf6305c4419" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } } }
I20260927 21:27:40.262802 8529 tablet_replica.cc:442] T ff085229dab2450680ba9e7028a3499d P f3c00d3e1aed43918fa13cba70ee76a6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:27:40.266304 8553 tablet_replica.cc:439] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8752312c7922404f8e197bf6305c4419" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } } }
I20260927 21:27:40.267207 8553 tablet_replica.cc:442] T ff085229dab2450680ba9e7028a3499d P 8752312c7922404f8e197bf6305c4419: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:27:40.270594 8529 tablet_replica.cc:439] T ff085229dab2450680ba9e7028a3499d P f3c00d3e1aed43918fa13cba70ee76a6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8752312c7922404f8e197bf6305c4419" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } } }
I20260927 21:27:40.270946 8529 tablet_replica.cc:442] T ff085229dab2450680ba9e7028a3499d P f3c00d3e1aed43918fa13cba70ee76a6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:27:40.302542 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:40.296036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35985 (local address 127.0.97.1:39273)
0927 21:27:40.296218 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:40.296224 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:40.296599 (+ 375us) server_negotiation.cc:407] Connection header received
0927 21:27:40.296866 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:40.296871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:40.296939 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:40.297050 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:27:40.297723 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.298539 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.300163 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.300667 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.300816 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:40.301318 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:27:40.301427 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:27:40.301552 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:40.302084 (+ 532us) server_negotiation.cc:299] Negotiation successful
0927 21:27:40.302170 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:27:40.302670 8563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:40.295569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39273 (local address 127.0.97.2:35985)
0927 21:27:40.296470 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:40.296487 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:40.296681 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:40.297070 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:40.297075 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:40.297088 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:27:40.297480 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:40.297490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.298701 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.298707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:40.300004 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:40.300017 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.300887 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.300894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:40.301058 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:40.301119 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:27:40.301941 (+ 822us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:27:40.301951 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 21:27:40.302060 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 21:27:40.302211 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":532,"threads_started":1}
I20260927 21:27:40.317889 8156 raft_consensus.cc:1261] T 31a917485f2140dab76ac82be7529969 P 8752312c7922404f8e197bf6305c4419 [term 1 FOLLOWER]: Refusing update from remote peer f3c00d3e1aed43918fa13cba70ee76a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:27:40.318778 8517 consensus_queue.cc:1035] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:27:40.323678 8441 raft_consensus.cc:1261] T 31a917485f2140dab76ac82be7529969 P 38123d87872d4302a64e23c254af75e6 [term 1 FOLLOWER]: Refusing update from remote peer f3c00d3e1aed43918fa13cba70ee76a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:27:40.324795 8517 consensus_queue.cc:1035] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:27:40.334081 8524 mvcc.cc:205] Tried to move back new op lower bound from 7334070109451653120 to 7334070108266475520. Current Snapshot: MvccSnapshot[applied={T|T < 7334070109451653120}]
I20260927 21:27:40.340375 8526 mvcc.cc:205] Tried to move back new op lower bound from 7334070109451653120 to 7334070108266475520. Current Snapshot: MvccSnapshot[applied={T|T < 7334070109451653120}]
I20260927 21:27:40.391055 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:40.383471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52301 (local address 127.0.97.2:40457)
0927 21:27:40.383649 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:40.383658 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:40.384539 (+ 881us) server_negotiation.cc:407] Connection header received
0927 21:27:40.385219 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:40.385227 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:40.385325 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:40.385381 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:27:40.386734 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.387445 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.388876 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.389483 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.389602 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:40.390253 (+ 651us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:27:40.390408 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:27:40.390473 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:40.390693 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 21:27:40.390785 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:27:40.391894 8569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:40.383511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40457 (local address 127.0.97.1:52301)
0927 21:27:40.384439 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:40.384460 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:40.385034 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:40.385942 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:40.385948 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:40.385961 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:27:40.386578 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:40.386589 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.387588 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.387596 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:40.388735 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:40.388746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.389792 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.389798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:40.389970 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:40.390055 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:27:40.390551 (+ 496us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:27:40.390563 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 21:27:40.391458 (+ 895us) client_negotiation.cc:240] Negotiation successful
0927 21:27:40.391603 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":328,"threads_started":1}
I20260927 21:27:40.444824 7997 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:27:40.445096 7997 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:27:40.448335 7997 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d517fba76edf42c8be94ff4072cb0616: Sending DeleteTablet for 3 replicas of tablet 31a917485f2140dab76ac82be7529969
I20260927 21:27:40.449566 8415 tablet_service.cc:1514] Processing DeleteTablet for tablet 31a917485f2140dab76ac82be7529969 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:27:40 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45644
I20260927 21:27:40.449880 8276 tablet_service.cc:1514] Processing DeleteTablet for tablet 31a917485f2140dab76ac82be7529969 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:27:40 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53132
I20260927 21:27:40.450536 8135 tablet_service.cc:1514] Processing DeleteTablet for tablet 31a917485f2140dab76ac82be7529969 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:27:40 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40936
I20260927 21:27:40.462227 8572 tablet_replica.cc:330] T 31a917485f2140dab76ac82be7529969 P 38123d87872d4302a64e23c254af75e6: stopping tablet replica
I20260927 21:27:40.462532 8572 raft_consensus.cc:2229] T 31a917485f2140dab76ac82be7529969 P 38123d87872d4302a64e23c254af75e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:27:40.462868 8572 raft_consensus.cc:2258] T 31a917485f2140dab76ac82be7529969 P 38123d87872d4302a64e23c254af75e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:27:40.464365 8572 ts_tablet_manager.cc:1905] T 31a917485f2140dab76ac82be7529969 P 38123d87872d4302a64e23c254af75e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:27:40.473431 8572 ts_tablet_manager.cc:1918] T 31a917485f2140dab76ac82be7529969 P 38123d87872d4302a64e23c254af75e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:27:40.473628 8572 log.cc:1196] T 31a917485f2140dab76ac82be7529969 P 38123d87872d4302a64e23c254af75e6: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-2/wal/wals/31a917485f2140dab76ac82be7529969
I20260927 21:27:40.474109 8572 ts_tablet_manager.cc:1939] T 31a917485f2140dab76ac82be7529969 P 38123d87872d4302a64e23c254af75e6: Deleting consensus metadata
I20260927 21:27:40.475455 7982 catalog_manager.cc:4841] TS 38123d87872d4302a64e23c254af75e6 (127.0.97.3:33407): tablet 31a917485f2140dab76ac82be7529969 (table test-table [id=337e476d3eb2415b9385553b3be688a2]) successfully deleted
I20260927 21:27:40.482409 8574 tablet_replica.cc:330] T 31a917485f2140dab76ac82be7529969 P 8752312c7922404f8e197bf6305c4419: stopping tablet replica
I20260927 21:27:40.482692 8574 raft_consensus.cc:2229] T 31a917485f2140dab76ac82be7529969 P 8752312c7922404f8e197bf6305c4419 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:27:40.483063 8574 raft_consensus.cc:2258] T 31a917485f2140dab76ac82be7529969 P 8752312c7922404f8e197bf6305c4419 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:27:40.483688 8573 tablet_replica.cc:330] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6: stopping tablet replica
I20260927 21:27:40.483901 8573 raft_consensus.cc:2229] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:27:40.484300 8573 raft_consensus.cc:2258] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:27:40.484611 8574 ts_tablet_manager.cc:1905] T 31a917485f2140dab76ac82be7529969 P 8752312c7922404f8e197bf6305c4419: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:27:40.485325 8573 ts_tablet_manager.cc:1905] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:27:40.492660 8573 ts_tablet_manager.cc:1918] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:27:40.492825 8573 log.cc:1196] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-1/wal/wals/31a917485f2140dab76ac82be7529969
I20260927 21:27:40.493273 8573 ts_tablet_manager.cc:1939] T 31a917485f2140dab76ac82be7529969 P f3c00d3e1aed43918fa13cba70ee76a6: Deleting consensus metadata
I20260927 21:27:40.494573 7984 catalog_manager.cc:4841] TS f3c00d3e1aed43918fa13cba70ee76a6 (127.0.97.2:40457): tablet 31a917485f2140dab76ac82be7529969 (table test-table [id=337e476d3eb2415b9385553b3be688a2]) successfully deleted
I20260927 21:27:40.495019 8574 ts_tablet_manager.cc:1918] T 31a917485f2140dab76ac82be7529969 P 8752312c7922404f8e197bf6305c4419: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:27:40.495187 8574 log.cc:1196] T 31a917485f2140dab76ac82be7529969 P 8752312c7922404f8e197bf6305c4419: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-0/wal/wals/31a917485f2140dab76ac82be7529969
I20260927 21:27:40.495728 8574 ts_tablet_manager.cc:1939] T 31a917485f2140dab76ac82be7529969 P 8752312c7922404f8e197bf6305c4419: Deleting consensus metadata
I20260927 21:27:40.496945 7985 catalog_manager.cc:4841] TS 8752312c7922404f8e197bf6305c4419 (127.0.97.1:39273): tablet 31a917485f2140dab76ac82be7529969 (table test-table [id=337e476d3eb2415b9385553b3be688a2]) successfully deleted
Sep 27 21:27:40 dist-test-slave-bxm0 krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:27:40 dist-test-slave-bxm0 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:40 dist-test-slave-bxm0 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 21:27:40.539695 8547 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:40.528953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48486 (local address 127.0.97.62:35055)
0927 21:27:40.529264 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:40.529273 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:40.530030 (+ 757us) server_negotiation.cc:407] Connection header received
0927 21:27:40.530259 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:40.530264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:40.530333 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:40.530447 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:40.531777 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.532554 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.533875 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.534273 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.534416 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:40.539494 (+ 5078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48486: BlockingRecv error: recv got EOF from 127.0.0.1:48486 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
Sep 27 21:27:40 dist-test-slave-bxm0 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 21:27:40.564730 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:40.547918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.0.97.62:35055)
0927 21:27:40.548137 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:40.548147 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:40.550218 (+ 2071us) server_negotiation.cc:407] Connection header received
0927 21:27:40.550456 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:40.550462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:40.550542 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:40.550656 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:40.551970 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.552695 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.554076 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.554498 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.554632 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:40.558848 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:40.558856 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:40.558861 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:40.558921 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:40.561986 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:40.562685 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:40.562691 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:40.562694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:40.562838 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:40.563229 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:40.563234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:40.563244 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:40.563544 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:40.563813 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:40.564106 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 21:27:40.564343 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 21:27:40.568854 7997 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:27:40.569492 7997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:27:40.583774 8135 tablet_service.cc:1467] Processing CreateTablet for tablet 8303119cdfa749e7979752c7b0b6acf9 (DEFAULT_TABLE table=test-table [id=274f428ff1a340449b9ce0142f5fd785]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:27:40.584255 8135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8303119cdfa749e7979752c7b0b6acf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:40.589188 8513 tablet_bootstrap.cc:492] T 8303119cdfa749e7979752c7b0b6acf9 P 8752312c7922404f8e197bf6305c4419: Bootstrap starting.
I20260927 21:27:40.589612 8276 tablet_service.cc:1467] Processing CreateTablet for tablet 8303119cdfa749e7979752c7b0b6acf9 (DEFAULT_TABLE table=test-table [id=274f428ff1a340449b9ce0142f5fd785]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:27:40.590374 8276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8303119cdfa749e7979752c7b0b6acf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:40.590935 8513 tablet_bootstrap.cc:654] T 8303119cdfa749e7979752c7b0b6acf9 P 8752312c7922404f8e197bf6305c4419: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:40.598966 8415 tablet_service.cc:1467] Processing CreateTablet for tablet 8303119cdfa749e7979752c7b0b6acf9 (DEFAULT_TABLE table=test-table [id=274f428ff1a340449b9ce0142f5fd785]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:27:40.599560 8415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8303119cdfa749e7979752c7b0b6acf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:40.602838 8513 tablet_bootstrap.cc:492] T 8303119cdfa749e7979752c7b0b6acf9 P 8752312c7922404f8e197bf6305c4419: No bootstrap required, opened a new log
I20260927 21:27:40.603075 8513 ts_tablet_manager.cc:1397] T 8303119cdfa749e7979752c7b0b6acf9 P 8752312c7922404f8e197bf6305c4419: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.003s
I20260927 21:27:40.603955 8513 raft_consensus.cc:348] T 8303119cdfa749e7979752c7b0b6acf9 P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:40.604399 8513 raft_consensus.cc:374] T 8303119cdfa749e7979752c7b0b6acf9 P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:40.604552 8513 raft_consensus.cc:729] T 8303119cdfa749e7979752c7b0b6acf9 P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8752312c7922404f8e197bf6305c4419, State: Initialized, Role: FOLLOWER
I20260927 21:27:40.604895 8513 consensus_queue.cc:260] T 8303119cdfa749e7979752c7b0b6acf9 P 8752312c7922404f8e197bf6305c4419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:40.605600 8513 ts_tablet_manager.cc:1428] T 8303119cdfa749e7979752c7b0b6acf9 P 8752312c7922404f8e197bf6305c4419: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:27:40.614197 8510 tablet_bootstrap.cc:492] T 8303119cdfa749e7979752c7b0b6acf9 P 38123d87872d4302a64e23c254af75e6: Bootstrap starting.
I20260927 21:27:40.615989 8510 tablet_bootstrap.cc:654] T 8303119cdfa749e7979752c7b0b6acf9 P 38123d87872d4302a64e23c254af75e6: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:40.623104 8511 tablet_bootstrap.cc:492] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6: Bootstrap starting.
I20260927 21:27:40.624806 8511 tablet_bootstrap.cc:654] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:40.634037 8510 tablet_bootstrap.cc:492] T 8303119cdfa749e7979752c7b0b6acf9 P 38123d87872d4302a64e23c254af75e6: No bootstrap required, opened a new log
I20260927 21:27:40.634311 8511 tablet_bootstrap.cc:492] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6: No bootstrap required, opened a new log
I20260927 21:27:40.634339 8510 ts_tablet_manager.cc:1397] T 8303119cdfa749e7979752c7b0b6acf9 P 38123d87872d4302a64e23c254af75e6: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.001s
I20260927 21:27:40.634473 8511 ts_tablet_manager.cc:1397] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.001s
I20260927 21:27:40.635264 8511 raft_consensus.cc:348] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:40.635269 8510 raft_consensus.cc:348] T 8303119cdfa749e7979752c7b0b6acf9 P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:40.635537 8511 raft_consensus.cc:374] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:40.635627 8511 raft_consensus.cc:729] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c00d3e1aed43918fa13cba70ee76a6, State: Initialized, Role: FOLLOWER
I20260927 21:27:40.635673 8510 raft_consensus.cc:374] T 8303119cdfa749e7979752c7b0b6acf9 P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:40.635879 8510 raft_consensus.cc:729] T 8303119cdfa749e7979752c7b0b6acf9 P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38123d87872d4302a64e23c254af75e6, State: Initialized, Role: FOLLOWER
I20260927 21:27:40.636230 8510 consensus_queue.cc:260] T 8303119cdfa749e7979752c7b0b6acf9 P 38123d87872d4302a64e23c254af75e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:40.636768 8511 consensus_queue.cc:260] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:40.637864 8510 ts_tablet_manager.cc:1428] T 8303119cdfa749e7979752c7b0b6acf9 P 38123d87872d4302a64e23c254af75e6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 21:27:40.638836 8511 ts_tablet_manager.cc:1428] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 21:27:40.658440 8517 raft_consensus.cc:482] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:27:40.658648 8517 raft_consensus.cc:504] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:40.659490 8517 leader_election.cc:288] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8752312c7922404f8e197bf6305c4419 (127.0.97.1:39273), 38123d87872d4302a64e23c254af75e6 (127.0.97.3:33407)
I20260927 21:27:40.660070 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8303119cdfa749e7979752c7b0b6acf9" candidate_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8752312c7922404f8e197bf6305c4419" is_pre_election: true
I20260927 21:27:40.660333 8156 raft_consensus.cc:2454] T 8303119cdfa749e7979752c7b0b6acf9 P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3c00d3e1aed43918fa13cba70ee76a6 in term 0.
I20260927 21:27:40.660291 8440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8303119cdfa749e7979752c7b0b6acf9" candidate_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38123d87872d4302a64e23c254af75e6" is_pre_election: true
I20260927 21:27:40.660567 8440 raft_consensus.cc:2454] T 8303119cdfa749e7979752c7b0b6acf9 P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3c00d3e1aed43918fa13cba70ee76a6 in term 0.
I20260927 21:27:40.660823 8229 leader_election.cc:302] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8752312c7922404f8e197bf6305c4419, f3c00d3e1aed43918fa13cba70ee76a6; no voters:
I20260927 21:27:40.661103 8517 raft_consensus.cc:2781] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:27:40.661216 8517 raft_consensus.cc:482] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:27:40.661321 8517 raft_consensus.cc:3037] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:40.663362 8517 raft_consensus.cc:504] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:40.664289 8517 leader_election.cc:288] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6 [CANDIDATE]: Term 1 election: Requested vote from peers 8752312c7922404f8e197bf6305c4419 (127.0.97.1:39273), 38123d87872d4302a64e23c254af75e6 (127.0.97.3:33407)
I20260927 21:27:40.664444 8440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8303119cdfa749e7979752c7b0b6acf9" candidate_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38123d87872d4302a64e23c254af75e6"
I20260927 21:27:40.664615 8440 raft_consensus.cc:3037] T 8303119cdfa749e7979752c7b0b6acf9 P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:40.667301 8440 raft_consensus.cc:2454] T 8303119cdfa749e7979752c7b0b6acf9 P 38123d87872d4302a64e23c254af75e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c00d3e1aed43918fa13cba70ee76a6 in term 1.
I20260927 21:27:40.667879 8226 leader_election.cc:302] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38123d87872d4302a64e23c254af75e6, f3c00d3e1aed43918fa13cba70ee76a6; no voters:
I20260927 21:27:40.667919 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8303119cdfa749e7979752c7b0b6acf9" candidate_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8752312c7922404f8e197bf6305c4419"
I20260927 21:27:40.668138 8156 raft_consensus.cc:3037] T 8303119cdfa749e7979752c7b0b6acf9 P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:40.668221 8517 raft_consensus.cc:2781] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:27:40.668696 8517 raft_consensus.cc:686] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6 [term 1 LEADER]: Becoming Leader. State: Replica: f3c00d3e1aed43918fa13cba70ee76a6, State: Running, Role: LEADER
I20260927 21:27:40.668998 8517 consensus_queue.cc:237] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:40.671841 8156 raft_consensus.cc:2454] T 8303119cdfa749e7979752c7b0b6acf9 P 8752312c7922404f8e197bf6305c4419 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c00d3e1aed43918fa13cba70ee76a6 in term 1.
I20260927 21:27:40.676647 7997 catalog_manager.cc:5495] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3c00d3e1aed43918fa13cba70ee76a6 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } health_report { overall_health: UNKNOWN } } }
I20260927 21:27:40.700963 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:40.695760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53148 (local address 127.0.97.2:40457)
0927 21:27:40.695964 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:40.695970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:40.696140 (+ 170us) server_negotiation.cc:407] Connection header received
0927 21:27:40.696393 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:40.696398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:40.696468 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:40.696569 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:27:40.696947 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.697784 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.699198 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:40.699650 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:40.699792 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:40.700196 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:27:40.700333 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:27:40.700447 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:40.700589 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 21:27:40.700670 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 21:27:40.704243 8440 raft_consensus.cc:1261] T 8303119cdfa749e7979752c7b0b6acf9 P 38123d87872d4302a64e23c254af75e6 [term 1 FOLLOWER]: Refusing update from remote peer f3c00d3e1aed43918fa13cba70ee76a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:27:40.704806 8156 raft_consensus.cc:1261] T 8303119cdfa749e7979752c7b0b6acf9 P 8752312c7922404f8e197bf6305c4419 [term 1 FOLLOWER]: Refusing update from remote peer f3c00d3e1aed43918fa13cba70ee76a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:27:40.705367 8517 consensus_queue.cc:1035] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:27:40.705657 8567 consensus_queue.cc:1035] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:27:40.730008 7996 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:27:40.730530 7996 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:27:40.734814 7996 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d517fba76edf42c8be94ff4072cb0616: Sending DeleteTablet for 3 replicas of tablet 8303119cdfa749e7979752c7b0b6acf9
I20260927 21:27:40.737092 8135 tablet_service.cc:1514] Processing DeleteTablet for tablet 8303119cdfa749e7979752c7b0b6acf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:27:40 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40936
I20260927 21:27:40.737432 8574 tablet_replica.cc:330] T 8303119cdfa749e7979752c7b0b6acf9 P 8752312c7922404f8e197bf6305c4419: stopping tablet replica
I20260927 21:27:40.737689 8574 raft_consensus.cc:2229] T 8303119cdfa749e7979752c7b0b6acf9 P 8752312c7922404f8e197bf6305c4419 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:27:40.737947 8574 raft_consensus.cc:2258] T 8303119cdfa749e7979752c7b0b6acf9 P 8752312c7922404f8e197bf6305c4419 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:27:40.739675 8416 tablet_service.cc:1514] Processing DeleteTablet for tablet 8303119cdfa749e7979752c7b0b6acf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:27:40 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45644
I20260927 21:27:40.739971 8572 tablet_replica.cc:330] T 8303119cdfa749e7979752c7b0b6acf9 P 38123d87872d4302a64e23c254af75e6: stopping tablet replica
I20260927 21:27:40.740155 8572 raft_consensus.cc:2229] T 8303119cdfa749e7979752c7b0b6acf9 P 38123d87872d4302a64e23c254af75e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:27:40.740406 8572 raft_consensus.cc:2258] T 8303119cdfa749e7979752c7b0b6acf9 P 38123d87872d4302a64e23c254af75e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:27:40.740757 8415 tablet_service.cc:1514] Processing DeleteTablet for tablet 8303119cdfa749e7979752c7b0b6acf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:27:40 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45644
I20260927 21:27:40.741168 8276 tablet_service.cc:1514] Processing DeleteTablet for tablet 8303119cdfa749e7979752c7b0b6acf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:27:40 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53132
I20260927 21:27:40.741585 7997 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 21:27:40.742269 8574 ts_tablet_manager.cc:1905] T 8303119cdfa749e7979752c7b0b6acf9 P 8752312c7922404f8e197bf6305c4419: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:27:40.742573 8573 tablet_replica.cc:330] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6: stopping tablet replica
I20260927 21:27:40.742787 8573 raft_consensus.cc:2229] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:27:40.743144 8573 raft_consensus.cc:2258] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 21:27:40.742372 7997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:27:40.747097 8572 ts_tablet_manager.cc:1905] T 8303119cdfa749e7979752c7b0b6acf9 P 38123d87872d4302a64e23c254af75e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:27:40.747856 8573 ts_tablet_manager.cc:1905] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:27:40.755208 8572 ts_tablet_manager.cc:1918] T 8303119cdfa749e7979752c7b0b6acf9 P 38123d87872d4302a64e23c254af75e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:27:40.755383 8572 log.cc:1196] T 8303119cdfa749e7979752c7b0b6acf9 P 38123d87872d4302a64e23c254af75e6: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-2/wal/wals/8303119cdfa749e7979752c7b0b6acf9
I20260927 21:27:40.755541 8276 tablet_service.cc:1467] Processing CreateTablet for tablet 5d6483e461f1452c8f9afc39cb19ff20 (DEFAULT_TABLE table=test-table [id=b8cbfd51894845c1bc0f20456cc483c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:27:40.755570 8135 tablet_service.cc:1467] Processing CreateTablet for tablet 5d6483e461f1452c8f9afc39cb19ff20 (DEFAULT_TABLE table=test-table [id=b8cbfd51894845c1bc0f20456cc483c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:27:40.755854 8572 ts_tablet_manager.cc:1939] T 8303119cdfa749e7979752c7b0b6acf9 P 38123d87872d4302a64e23c254af75e6: Deleting consensus metadata
I20260927 21:27:40.756017 8135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d6483e461f1452c8f9afc39cb19ff20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:40.756013 8276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d6483e461f1452c8f9afc39cb19ff20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:40.760210 7982 catalog_manager.cc:4841] TS 38123d87872d4302a64e23c254af75e6 (127.0.97.3:33407): tablet 8303119cdfa749e7979752c7b0b6acf9 (table test-table [id=274f428ff1a340449b9ce0142f5fd785]) successfully deleted
W20260927 21:27:40.760684 7982 catalog_manager.cc:4808] TS 38123d87872d4302a64e23c254af75e6 (127.0.97.3:33407): delete failed for tablet 8303119cdfa749e7979752c7b0b6acf9 because the tablet was not found. No further retry: Not found: Tablet not found: 8303119cdfa749e7979752c7b0b6acf9
I20260927 21:27:40.760633 8415 tablet_service.cc:1467] Processing CreateTablet for tablet 5d6483e461f1452c8f9afc39cb19ff20 (DEFAULT_TABLE table=test-table [id=b8cbfd51894845c1bc0f20456cc483c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:27:40.761045 8415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d6483e461f1452c8f9afc39cb19ff20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:40.762571 8511 tablet_bootstrap.cc:492] T 5d6483e461f1452c8f9afc39cb19ff20 P f3c00d3e1aed43918fa13cba70ee76a6: Bootstrap starting.
I20260927 21:27:40.764248 8511 tablet_bootstrap.cc:654] T 5d6483e461f1452c8f9afc39cb19ff20 P f3c00d3e1aed43918fa13cba70ee76a6: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:40.767464 8510 tablet_bootstrap.cc:492] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6: Bootstrap starting.
I20260927 21:27:40.769065 8510 tablet_bootstrap.cc:654] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:40.771186 8513 tablet_bootstrap.cc:492] T 5d6483e461f1452c8f9afc39cb19ff20 P 8752312c7922404f8e197bf6305c4419: Bootstrap starting.
I20260927 21:27:40.771246 8510 tablet_bootstrap.cc:492] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6: No bootstrap required, opened a new log
I20260927 21:27:40.771399 8510 ts_tablet_manager.cc:1397] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260927 21:27:40.772135 8510 raft_consensus.cc:348] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:40.772367 8510 raft_consensus.cc:374] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:40.772429 8510 raft_consensus.cc:729] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38123d87872d4302a64e23c254af75e6, State: Initialized, Role: FOLLOWER
I20260927 21:27:40.772711 8510 consensus_queue.cc:260] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:40.773033 8513 tablet_bootstrap.cc:654] T 5d6483e461f1452c8f9afc39cb19ff20 P 8752312c7922404f8e197bf6305c4419: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:40.774152 8573 ts_tablet_manager.cc:1918] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:27:40.774322 8573 log.cc:1196] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-1/wal/wals/8303119cdfa749e7979752c7b0b6acf9
I20260927 21:27:40.774744 8573 ts_tablet_manager.cc:1939] T 8303119cdfa749e7979752c7b0b6acf9 P f3c00d3e1aed43918fa13cba70ee76a6: Deleting consensus metadata
I20260927 21:27:40.775853 7984 catalog_manager.cc:4841] TS f3c00d3e1aed43918fa13cba70ee76a6 (127.0.97.2:40457): tablet 8303119cdfa749e7979752c7b0b6acf9 (table test-table [id=274f428ff1a340449b9ce0142f5fd785]) successfully deleted
I20260927 21:27:40.776125 8510 ts_tablet_manager.cc:1428] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260927 21:27:40.779459 8574 ts_tablet_manager.cc:1918] T 8303119cdfa749e7979752c7b0b6acf9 P 8752312c7922404f8e197bf6305c4419: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:27:40.779651 8574 log.cc:1196] T 8303119cdfa749e7979752c7b0b6acf9 P 8752312c7922404f8e197bf6305c4419: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-0/wal/wals/8303119cdfa749e7979752c7b0b6acf9
I20260927 21:27:40.780161 8574 ts_tablet_manager.cc:1939] T 8303119cdfa749e7979752c7b0b6acf9 P 8752312c7922404f8e197bf6305c4419: Deleting consensus metadata
I20260927 21:27:40.781292 7985 catalog_manager.cc:4841] TS 8752312c7922404f8e197bf6305c4419 (127.0.97.1:39273): tablet 8303119cdfa749e7979752c7b0b6acf9 (table test-table [id=274f428ff1a340449b9ce0142f5fd785]) successfully deleted
I20260927 21:27:40.786505 8513 tablet_bootstrap.cc:492] T 5d6483e461f1452c8f9afc39cb19ff20 P 8752312c7922404f8e197bf6305c4419: No bootstrap required, opened a new log
I20260927 21:27:40.786578 8511 tablet_bootstrap.cc:492] T 5d6483e461f1452c8f9afc39cb19ff20 P f3c00d3e1aed43918fa13cba70ee76a6: No bootstrap required, opened a new log
I20260927 21:27:40.786736 8513 ts_tablet_manager.cc:1397] T 5d6483e461f1452c8f9afc39cb19ff20 P 8752312c7922404f8e197bf6305c4419: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.007s
I20260927 21:27:40.786784 8511 ts_tablet_manager.cc:1397] T 5d6483e461f1452c8f9afc39cb19ff20 P f3c00d3e1aed43918fa13cba70ee76a6: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.007s
I20260927 21:27:40.787418 8513 raft_consensus.cc:348] T 5d6483e461f1452c8f9afc39cb19ff20 P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:40.787652 8513 raft_consensus.cc:374] T 5d6483e461f1452c8f9afc39cb19ff20 P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:40.787619 8511 raft_consensus.cc:348] T 5d6483e461f1452c8f9afc39cb19ff20 P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:40.787765 8513 raft_consensus.cc:729] T 5d6483e461f1452c8f9afc39cb19ff20 P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8752312c7922404f8e197bf6305c4419, State: Initialized, Role: FOLLOWER
I20260927 21:27:40.787886 8511 raft_consensus.cc:374] T 5d6483e461f1452c8f9afc39cb19ff20 P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:40.787973 8511 raft_consensus.cc:729] T 5d6483e461f1452c8f9afc39cb19ff20 P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c00d3e1aed43918fa13cba70ee76a6, State: Initialized, Role: FOLLOWER
I20260927 21:27:40.788059 8513 consensus_queue.cc:260] T 5d6483e461f1452c8f9afc39cb19ff20 P 8752312c7922404f8e197bf6305c4419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:40.788229 8511 consensus_queue.cc:260] T 5d6483e461f1452c8f9afc39cb19ff20 P f3c00d3e1aed43918fa13cba70ee76a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:40.788640 8513 ts_tablet_manager.cc:1428] T 5d6483e461f1452c8f9afc39cb19ff20 P 8752312c7922404f8e197bf6305c4419: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260927 21:27:40.788796 8511 ts_tablet_manager.cc:1428] T 5d6483e461f1452c8f9afc39cb19ff20 P f3c00d3e1aed43918fa13cba70ee76a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 21:27:41.115794 8516 raft_consensus.cc:482] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:27:41.116041 8516 raft_consensus.cc:504] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:41.117573 8516 leader_election.cc:288] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8752312c7922404f8e197bf6305c4419 (127.0.97.1:39273), f3c00d3e1aed43918fa13cba70ee76a6 (127.0.97.2:40457)
I20260927 21:27:41.124933 8599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:41.118018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47569 (local address 127.0.97.1:39273)
0927 21:27:41.118578 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:41.118582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:41.118778 (+ 196us) server_negotiation.cc:407] Connection header received
0927 21:27:41.119056 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:41.119061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:41.119151 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:41.119277 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:41.119920 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:41.120777 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:41.123010 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:41.123926 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:41.124035 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:41.124108 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:41.124480 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 21:27:41.124637 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":369,"threads_started":1}
I20260927 21:27:41.124954 8598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:41.117928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39273 (local address 127.0.97.3:47569)
0927 21:27:41.118625 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:41.118656 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:41.118867 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:41.119341 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:41.119349 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:41.119384 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:41.119684 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:41.119706 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:41.120929 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:41.120935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:41.122832 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:41.122847 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:41.124103 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:41.124109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:41.124289 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:41.124324 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 21:27:41.124425 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 21:27:41.124519 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":77,"thread_start_us":341,"threads_started":1}
I20260927 21:27:41.125036 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:41.118019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51161 (local address 127.0.97.2:40457)
0927 21:27:41.118223 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:41.118233 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:41.119314 (+ 1081us) server_negotiation.cc:407] Connection header received
0927 21:27:41.119471 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:41.119477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:41.119562 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:41.119679 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:27:41.120280 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:41.121100 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:41.123172 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:41.124098 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:41.124222 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:41.124322 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:41.124614 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 21:27:41.124749 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:27:41.125097 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:41.117928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40457 (local address 127.0.97.3:51161)
0927 21:27:41.119189 (+ 1261us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:41.119209 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:41.119330 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:41.119691 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:41.119695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:41.119713 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:27:41.120092 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:41.120127 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:41.121307 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:41.121311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:41.123003 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:41.123018 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:41.124295 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:41.124299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:41.124446 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:41.124477 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 21:27:41.124568 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 21:27:41.124681 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1144,"mutex_wait_us":34,"thread_start_us":949,"threads_started":1}
I20260927 21:27:41.125825 8301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d6483e461f1452c8f9afc39cb19ff20" candidate_uuid: "38123d87872d4302a64e23c254af75e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" is_pre_election: true
I20260927 21:27:41.126144 8301 raft_consensus.cc:2454] T 5d6483e461f1452c8f9afc39cb19ff20 P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38123d87872d4302a64e23c254af75e6 in term 0.
I20260927 21:27:41.126065 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d6483e461f1452c8f9afc39cb19ff20" candidate_uuid: "38123d87872d4302a64e23c254af75e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8752312c7922404f8e197bf6305c4419" is_pre_election: true
I20260927 21:27:41.126338 8156 raft_consensus.cc:2454] T 5d6483e461f1452c8f9afc39cb19ff20 P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38123d87872d4302a64e23c254af75e6 in term 0.
I20260927 21:27:41.126648 8368 leader_election.cc:302] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38123d87872d4302a64e23c254af75e6, f3c00d3e1aed43918fa13cba70ee76a6; no voters:
I20260927 21:27:41.127051 8516 raft_consensus.cc:2781] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:27:41.127184 8516 raft_consensus.cc:482] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:27:41.127283 8516 raft_consensus.cc:3037] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:41.129464 8516 raft_consensus.cc:504] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:41.130170 8516 leader_election.cc:288] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6 [CANDIDATE]: Term 1 election: Requested vote from peers 8752312c7922404f8e197bf6305c4419 (127.0.97.1:39273), f3c00d3e1aed43918fa13cba70ee76a6 (127.0.97.2:40457)
I20260927 21:27:41.130606 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d6483e461f1452c8f9afc39cb19ff20" candidate_uuid: "38123d87872d4302a64e23c254af75e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8752312c7922404f8e197bf6305c4419"
I20260927 21:27:41.130833 8156 raft_consensus.cc:3037] T 5d6483e461f1452c8f9afc39cb19ff20 P 8752312c7922404f8e197bf6305c4419 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:41.130746 8301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d6483e461f1452c8f9afc39cb19ff20" candidate_uuid: "38123d87872d4302a64e23c254af75e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c00d3e1aed43918fa13cba70ee76a6"
I20260927 21:27:41.130986 8301 raft_consensus.cc:3037] T 5d6483e461f1452c8f9afc39cb19ff20 P f3c00d3e1aed43918fa13cba70ee76a6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:41.132930 8301 raft_consensus.cc:2454] T 5d6483e461f1452c8f9afc39cb19ff20 P f3c00d3e1aed43918fa13cba70ee76a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38123d87872d4302a64e23c254af75e6 in term 1.
I20260927 21:27:41.132918 8156 raft_consensus.cc:2454] T 5d6483e461f1452c8f9afc39cb19ff20 P 8752312c7922404f8e197bf6305c4419 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38123d87872d4302a64e23c254af75e6 in term 1.
I20260927 21:27:41.133358 8368 leader_election.cc:302] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38123d87872d4302a64e23c254af75e6, f3c00d3e1aed43918fa13cba70ee76a6; no voters:
I20260927 21:27:41.133684 8516 raft_consensus.cc:2781] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:27:41.134199 8516 raft_consensus.cc:686] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6 [term 1 LEADER]: Becoming Leader. State: Replica: 38123d87872d4302a64e23c254af75e6, State: Running, Role: LEADER
I20260927 21:27:41.134599 8516 consensus_queue.cc:237] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } }
I20260927 21:27:41.138346 7996 catalog_manager.cc:5495] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38123d87872d4302a64e23c254af75e6 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "38123d87872d4302a64e23c254af75e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38123d87872d4302a64e23c254af75e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33407 } health_report { overall_health: HEALTHY } } }
I20260927 21:27:41.254225 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:41.245459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45654 (local address 127.0.97.3:33407)
0927 21:27:41.248157 (+ 2698us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:41.248161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:41.248181 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:27:41.248266 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:41.248269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:41.248350 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:41.248491 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:27:41.248984 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:41.249875 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:41.251841 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:41.252246 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:41.252345 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:41.252932 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:27:41.253514 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:27:41.253692 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:41.253790 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 21:27:41.253867 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2596,"thread_start_us":2460,"threads_started":1}
I20260927 21:27:41.269765 8599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:41.264314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52783 (local address 127.0.97.1:39273)
0927 21:27:41.264496 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:41.264501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:41.264976 (+ 475us) server_negotiation.cc:407] Connection header received
0927 21:27:41.265184 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:41.265188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:41.265265 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:41.265376 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:27:41.265913 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:41.266662 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:41.268073 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:41.268473 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:41.268601 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:41.269056 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:27:41.269163 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:27:41.269268 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:41.269365 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 21:27:41.269440 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:27:41.269883 8613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:41.264262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39273 (local address 127.0.97.3:52783)
0927 21:27:41.264884 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:41.264900 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:41.265018 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:41.265393 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:41.265398 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:41.265409 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:27:41.265762 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:41.265772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:41.266838 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:41.266844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:41.267929 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:41.267941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:41.268659 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:41.268667 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:41.268817 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:41.268871 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:27:41.269260 (+ 389us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:27:41.269270 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 21:27:41.269347 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 21:27:41.269480 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":324,"threads_started":1}
I20260927 21:27:41.278738 8301 raft_consensus.cc:1261] T 5d6483e461f1452c8f9afc39cb19ff20 P f3c00d3e1aed43918fa13cba70ee76a6 [term 1 FOLLOWER]: Refusing update from remote peer 38123d87872d4302a64e23c254af75e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:27:41.278738 8156 raft_consensus.cc:1261] T 5d6483e461f1452c8f9afc39cb19ff20 P 8752312c7922404f8e197bf6305c4419 [term 1 FOLLOWER]: Refusing update from remote peer 38123d87872d4302a64e23c254af75e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:27:41.279419 8516 consensus_queue.cc:1035] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8752312c7922404f8e197bf6305c4419" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:27:41.279711 8600 consensus_queue.cc:1035] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c00d3e1aed43918fa13cba70ee76a6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:27:41.306977 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:41.299322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56195 (local address 127.0.97.3:33407)
0927 21:27:41.299509 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:41.299514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:41.302167 (+ 2653us) server_negotiation.cc:407] Connection header received
0927 21:27:41.302285 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:41.302290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:41.302369 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:41.302500 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:27:41.302902 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:41.303821 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:41.305143 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:41.305697 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:41.305809 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:41.306250 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:27:41.306378 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:27:41.306474 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:41.306608 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 21:27:41.306668 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 21:27:41.307090 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:41.299242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33407 (local address 127.0.97.1:56195)
0927 21:27:41.302067 (+ 2825us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:41.302087 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:41.302189 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:41.302504 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:41.302507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:41.302516 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:27:41.302753 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:41.302766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:41.303953 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:41.303960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:41.305001 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:41.305012 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:41.305872 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:41.305877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:41.306044 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:41.306095 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:27:41.306487 (+ 392us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:27:41.306498 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 21:27:41.306585 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 21:27:41.306714 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2693,"thread_start_us":2570,"threads_started":1}
I20260927 21:27:41.337524 7996 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:27:41.337806 7996 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:27:41.340938 7996 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d517fba76edf42c8be94ff4072cb0616: Sending DeleteTablet for 3 replicas of tablet 5d6483e461f1452c8f9afc39cb19ff20
I20260927 21:27:41.341821 8276 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d6483e461f1452c8f9afc39cb19ff20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:27:41 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53132
I20260927 21:27:41.341877 8135 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d6483e461f1452c8f9afc39cb19ff20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:27:41 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40936
I20260927 21:27:41.342372 8415 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d6483e461f1452c8f9afc39cb19ff20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:27:41 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45644
I20260927 21:27:41.342788 8622 tablet_replica.cc:330] T 5d6483e461f1452c8f9afc39cb19ff20 P 8752312c7922404f8e197bf6305c4419: stopping tablet replica
I20260927 21:27:41.343029 8622 raft_consensus.cc:2229] T 5d6483e461f1452c8f9afc39cb19ff20 P 8752312c7922404f8e197bf6305c4419 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:27:41.343128 8623 tablet_replica.cc:330] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6: stopping tablet replica
I20260927 21:27:41.343319 8623 raft_consensus.cc:2229] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:27:41.344151 8623 raft_consensus.cc:2258] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:27:41.345510 8623 ts_tablet_manager.cc:1905] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:27:41.346732 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 8070
I20260927 21:27:41.351809 8621 tablet_replica.cc:330] T 5d6483e461f1452c8f9afc39cb19ff20 P f3c00d3e1aed43918fa13cba70ee76a6: stopping tablet replica
I20260927 21:27:41.352141 8621 raft_consensus.cc:2229] T 5d6483e461f1452c8f9afc39cb19ff20 P f3c00d3e1aed43918fa13cba70ee76a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:27:41.352393 8621 raft_consensus.cc:2258] T 5d6483e461f1452c8f9afc39cb19ff20 P f3c00d3e1aed43918fa13cba70ee76a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:27:41.353340 8621 ts_tablet_manager.cc:1905] T 5d6483e461f1452c8f9afc39cb19ff20 P f3c00d3e1aed43918fa13cba70ee76a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:27:41.354281 8622 raft_consensus.cc:2258] T 5d6483e461f1452c8f9afc39cb19ff20 P 8752312c7922404f8e197bf6305c4419 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:27:41.354875 8623 ts_tablet_manager.cc:1918] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:27:41.355031 8623 log.cc:1196] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-2/wal/wals/5d6483e461f1452c8f9afc39cb19ff20
I20260927 21:27:41.355516 8623 ts_tablet_manager.cc:1939] T 5d6483e461f1452c8f9afc39cb19ff20 P 38123d87872d4302a64e23c254af75e6: Deleting consensus metadata
I20260927 21:27:41.356703 7982 catalog_manager.cc:4841] TS 38123d87872d4302a64e23c254af75e6 (127.0.97.3:33407): tablet 5d6483e461f1452c8f9afc39cb19ff20 (table test-table [id=b8cbfd51894845c1bc0f20456cc483c8]) successfully deleted
I20260927 21:27:41.359673 8621 ts_tablet_manager.cc:1918] T 5d6483e461f1452c8f9afc39cb19ff20 P f3c00d3e1aed43918fa13cba70ee76a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:27:41.359820 8621 log.cc:1196] T 5d6483e461f1452c8f9afc39cb19ff20 P f3c00d3e1aed43918fa13cba70ee76a6: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-1/wal/wals/5d6483e461f1452c8f9afc39cb19ff20
I20260927 21:27:41.360231 8621 ts_tablet_manager.cc:1939] T 5d6483e461f1452c8f9afc39cb19ff20 P f3c00d3e1aed43918fa13cba70ee76a6: Deleting consensus metadata
I20260927 21:27:41.361310 7984 catalog_manager.cc:4841] TS f3c00d3e1aed43918fa13cba70ee76a6 (127.0.97.2:40457): tablet 5d6483e461f1452c8f9afc39cb19ff20 (table test-table [id=b8cbfd51894845c1bc0f20456cc483c8]) successfully deleted
I20260927 21:27:41.361790 8622 ts_tablet_manager.cc:1905] T 5d6483e461f1452c8f9afc39cb19ff20 P 8752312c7922404f8e197bf6305c4419: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:27:41.367403 8622 ts_tablet_manager.cc:1918] T 5d6483e461f1452c8f9afc39cb19ff20 P 8752312c7922404f8e197bf6305c4419: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:27:41.367556 8622 log.cc:1196] T 5d6483e461f1452c8f9afc39cb19ff20 P 8752312c7922404f8e197bf6305c4419: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790544390748963-388-0/minicluster-data/ts-0/wal/wals/5d6483e461f1452c8f9afc39cb19ff20
I20260927 21:27:41.367936 8622 ts_tablet_manager.cc:1939] T 5d6483e461f1452c8f9afc39cb19ff20 P 8752312c7922404f8e197bf6305c4419: Deleting consensus metadata
I20260927 21:27:41.368978 7985 catalog_manager.cc:4841] TS 8752312c7922404f8e197bf6305c4419 (127.0.97.1:39273): tablet 5d6483e461f1452c8f9afc39cb19ff20 (table test-table [id=b8cbfd51894845c1bc0f20456cc483c8]) successfully deleted
Sep 27 21:27:41 dist-test-slave-bxm0 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260927 21:27:41.376677 8599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:41.353932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45138 (local address 127.0.97.1:39273)
0927 21:27:41.354142 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:41.354148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:41.359242 (+ 5094us) server_negotiation.cc:407] Connection header received
0927 21:27:41.359510 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:41.359516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:41.359604 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:41.359727 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:41.361338 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:41.362520 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:41.364457 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:41.365241 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:41.365407 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:41.370462 (+ 5055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:41.370477 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:41.370494 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:41.370532 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:41.373389 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:41.374207 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:41.374224 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:41.374226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:41.374606 (+ 380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:41.375060 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:41.375069 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:41.375071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:41.375449 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:41.375757 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:41.376066 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 21:27:41.376294 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 21:27:41.377334 8201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:41.519599 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 8070
I20260927 21:27:41.542979 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 8210
Sep 27 21:27:41 dist-test-slave-bxm0 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260927 21:27:41.559445 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:41.543809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50082 (local address 127.0.97.2:40457)
0927 21:27:41.543996 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:41.544004 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:41.544026 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:27:41.544128 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:41.544134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:41.544218 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:41.544345 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:41.545512 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:41.546677 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:41.547856 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:41.548519 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:41.548651 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:41.553225 (+ 4574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:41.553243 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:41.553259 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:41.553300 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:41.556432 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:41.557172 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:41.557182 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:41.557184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:41.557322 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:41.557775 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:41.557781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:41.557785 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:41.558201 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:41.558529 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:41.558829 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 21:27:41.559050 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 21:27:41.560308 8341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:41.689111 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 8210
I20260927 21:27:41.714582 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 8350
Sep 27 21:27:41 dist-test-slave-bxm0 krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260927 21:27:41.736104 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:41.715426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47790 (local address 127.0.97.3:33407)
0927 21:27:41.715659 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:41.715670 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:41.715799 (+ 129us) server_negotiation.cc:407] Connection header received
0927 21:27:41.715991 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:41.715996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:41.716088 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:41.716199 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:41.717959 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:41.719195 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:41.721184 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:41.722178 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:41.722315 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:41.728107 (+ 5792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:41.728135 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:41.728157 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:41.728208 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:41.732698 (+ 4490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:41.733652 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:41.733662 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:41.733665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:41.733832 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:41.734248 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:41.734259 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:41.734262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:41.734781 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:41.735083 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:41.735487 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 21:27:41.735734 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:27:41.736970 8481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:41.872786 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 8350
I20260927 21:27:41.894590 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 7966
I20260927 21:27:41.895872 8038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:42.009004 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 7966
2026-09-27T21:27:42Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6200 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:27:42 dist-test-slave-bxm0 krb5kdc[8631](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:27:42 dist-test-slave-bxm0 krb5kdc[8631](info): set up 2 sockets
Sep 27 21:27:42 dist-test-slave-bxm0 krb5kdc[8631](info): commencing operation
krb5kdc: starting...
W20260927 21:27:43.114629 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:27:43 dist-test-slave-bxm0 krb5kdc[8631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:27:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:27:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:27:43.341380 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38617
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:43.670848 8644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:43.671155 8644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:43.671321 8644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:43.678776 8644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:27:43.678905 8644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:43.678992 8644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:43.679096 8644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:27:43.679178 8644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:27:43.689250 8644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:43.690611 8644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:43.691967 8644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:43.706142 8649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:43.706177 8650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:43.706388 8652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:43.707202 8644 server_base.cc:1034] running on GCE node
I20260927 21:27:43.708098 8644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:43.709601 8644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:43.710906 8644 hybrid_clock.cc:648] HybridClock initialized: now 1790544463710845 us; error 42 us; skew 500 ppm
I20260927 21:27:43.711259 8644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T21:27:43Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1768 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:27:43 dist-test-slave-bxm0 krb5kdc[8658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:27:43 dist-test-slave-bxm0 krb5kdc[8658](info): set up 2 sockets
Sep 27 21:27:43 dist-test-slave-bxm0 krb5kdc[8658](info): commencing operation
krb5kdc: starting...
W20260927 21:27:44.882714 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:27:45 dist-test-slave-bxm0 krb5kdc[8658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:27:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:27:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:27:45.113464 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41753
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:45.443437 8671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:45.443742 8671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:45.443840 8671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:45.451061 8671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:27:45.451184 8671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:45.451236 8671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:45.451292 8671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:27:45.451336 8671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:27:45.462020 8671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:45.463470 8671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:45.464748 8671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:45.477288 8676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:45.477377 8677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:45.477339 8679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:45.478641 8671 server_base.cc:1034] running on GCE node
I20260927 21:27:45.479527 8671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:45.481094 8671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:45.482347 8671 hybrid_clock.cc:648] HybridClock initialized: now 1790544465482313 us; error 55 us; skew 500 ppm
I20260927 21:27:45.482719 8671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:45 dist-test-slave-bxm0 krb5kdc[8658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544465, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:45.490948 8671 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:27:45.492724 8671 webserver.cc:460] Webserver started at http://127.0.97.62:41659/ using document root <none> and password file <none>
I20260927 21:27:45.493507 8671 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:45.493698 8671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:45.494057 8671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:45.497349 8671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "ff4e61de07784def89452725e0f3bb2d"
format_stamp: "Formatted at 2026-09-27 21:27:45 on dist-test-slave-bxm0"
server_key: "4e5f4ef65c4d548e05bd86aef52bf8ee"
server_key_iv: "5a844bebf420b16aa5a116c7cab11819"
server_key_version: "encryptionkey@0"
I20260927 21:27:45.498343 8671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "ff4e61de07784def89452725e0f3bb2d"
format_stamp: "Formatted at 2026-09-27 21:27:45 on dist-test-slave-bxm0"
server_key: "4e5f4ef65c4d548e05bd86aef52bf8ee"
server_key_iv: "5a844bebf420b16aa5a116c7cab11819"
server_key_version: "encryptionkey@0"
I20260927 21:27:45.504093 8671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:27:45.508078 8686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:45.509728 8671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:27:45.509963 8671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "ff4e61de07784def89452725e0f3bb2d"
format_stamp: "Formatted at 2026-09-27 21:27:45 on dist-test-slave-bxm0"
server_key: "4e5f4ef65c4d548e05bd86aef52bf8ee"
server_key_iv: "5a844bebf420b16aa5a116c7cab11819"
server_key_version: "encryptionkey@0"
I20260927 21:27:45.510200 8671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:45.527870 8671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:45.532089 8671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:45.532409 8671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:45.551548 8671 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41753
I20260927 21:27:45.551577 8737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41753 every 8 connection(s)
I20260927 21:27:45.553071 8671 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:27:45.555670 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 8671
I20260927 21:27:45.555920 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:27:45.556573 388 external_mini_cluster.cc:1428] Setting key 647564dc76677ea42f97ac84df01d2c4
I20260927 21:27:45.557163 8738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:27:45.570147 8738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d: Bootstrap starting.
Sep 27 21:27:45 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:27:45.573822 8738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d: Neither blocks nor log segments found. Creating new log.
I20260927 21:27:45.574915 8738 log.cc:824] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d: Log is configured to *not* fsync() on all Append() calls
I20260927 21:27:45.577909 8738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d: No bootstrap required, opened a new log
I20260927 21:27:45.580251 8741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:45.558677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48182 (local address 127.0.97.62:41753)
0927 21:27:45.559461 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:45.559474 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:45.559514 (+ 40us) server_negotiation.cc:407] Connection header received
0927 21:27:45.562083 (+ 2569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:45.562110 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:45.562589 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:45.563044 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:45.564593 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:45.565933 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:45.567172 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:45.567906 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:45.568147 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:45.572372 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:45.572381 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:45.572397 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:45.572442 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:45.575737 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:45.576467 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:45.576475 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:45.576482 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:45.576644 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:45.577036 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:45.577041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:45.577044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:45.577573 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:45.577925 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:45.578400 (+ 475us) server_negotiation.cc:299] Negotiation successful
0927 21:27:45.578890 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":341,"threads_started":1}
I20260927 21:27:45.586773 8738 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4e61de07784def89452725e0f3bb2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41753 } }
I20260927 21:27:45.587147 8738 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:27:45.587291 8738 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff4e61de07784def89452725e0f3bb2d, State: Initialized, Role: FOLLOWER
I20260927 21:27:45.587951 8738 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4e61de07784def89452725e0f3bb2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41753 } }
I20260927 21:27:45.588183 8738 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:27:45.588313 8738 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:27:45.588476 8738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:27:45.590711 8738 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4e61de07784def89452725e0f3bb2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41753 } }
I20260927 21:27:45.591229 8738 leader_election.cc:302] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff4e61de07784def89452725e0f3bb2d; no voters:
I20260927 21:27:45.591813 8738 leader_election.cc:288] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:27:45.591962 8743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:27:45.593161 8743 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [term 1 LEADER]: Becoming Leader. State: Replica: ff4e61de07784def89452725e0f3bb2d, State: Running, Role: LEADER
I20260927 21:27:45.593815 8743 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4e61de07784def89452725e0f3bb2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41753 } }
I20260927 21:27:45.594135 8738 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:27:45.596647 8744 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff4e61de07784def89452725e0f3bb2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4e61de07784def89452725e0f3bb2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41753 } } }
I20260927 21:27:45.596697 8745 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff4e61de07784def89452725e0f3bb2d. Latest consensus state: current_term: 1 leader_uuid: "ff4e61de07784def89452725e0f3bb2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4e61de07784def89452725e0f3bb2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41753 } } }
I20260927 21:27:45.597165 8745 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:45.597165 8744 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:45.598215 8751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:27:45.604139 8751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:27:45.613574 8751 catalog_manager.cc:1347] Generated new cluster ID: 867765f7cb3b487ea36e3a52d2ba7775
I20260927 21:27:45.613751 8751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:27:45.645516 8751 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:27:45.646911 8751 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:27:45.654646 8751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d: Generated new TSK 0
I20260927 21:27:45.655539 8751 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 21:27:45.721544 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41753
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:46.057482 8764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:46.057808 8764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:46.057996 8764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:46.065794 8764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:46.065907 8764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:46.066071 8764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:27:46.076629 8764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:46.078035 8764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:46.079371 8764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:46.092909 8769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:46.092950 8770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:46.092978 8772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:46.094081 8764 server_base.cc:1034] running on GCE node
I20260927 21:27:46.094805 8764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:46.095942 8764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:46.097170 8764 hybrid_clock.cc:648] HybridClock initialized: now 1790544466097096 us; error 59 us; skew 500 ppm
I20260927 21:27:46.097482 8764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:46 dist-test-slave-bxm0 krb5kdc[8658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:46.100870 8764 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 21:27:46.102483 8764 webserver.cc:460] Webserver started at http://127.0.97.1:37047/ using document root <none> and password file <none>
I20260927 21:27:46.103178 8764 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:46.103293 8764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:46.103543 8764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:46.106561 8764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/data/instance:
uuid: "5c1182f9024c4cf19e2e2457d5cef552"
format_stamp: "Formatted at 2026-09-27 21:27:46 on dist-test-slave-bxm0"
server_key: "cdf33c08a45b90c6b5f8c489163d6cf2"
server_key_iv: "71471d3b35a1aa23b70d77dd8c32cd00"
server_key_version: "encryptionkey@0"
I20260927 21:27:46.107345 8764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5c1182f9024c4cf19e2e2457d5cef552"
format_stamp: "Formatted at 2026-09-27 21:27:46 on dist-test-slave-bxm0"
server_key: "cdf33c08a45b90c6b5f8c489163d6cf2"
server_key_iv: "71471d3b35a1aa23b70d77dd8c32cd00"
server_key_version: "encryptionkey@0"
I20260927 21:27:46.112701 8764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 21:27:46.116374 8779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:46.117908 8764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:27:46.118050 8764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "5c1182f9024c4cf19e2e2457d5cef552"
format_stamp: "Formatted at 2026-09-27 21:27:46 on dist-test-slave-bxm0"
server_key: "cdf33c08a45b90c6b5f8c489163d6cf2"
server_key_iv: "71471d3b35a1aa23b70d77dd8c32cd00"
server_key_version: "encryptionkey@0"
I20260927 21:27:46.118232 8764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:46.135738 8764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:46.140352 8764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:46.140612 8764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:46.141557 8764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:27:46.142954 8764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:46.143028 8764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:46.143266 8764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:46.143354 8764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:46.177322 8764 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44145
I20260927 21:27:46.177372 8891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44145 every 8 connection(s)
I20260927 21:27:46.178695 8764 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:27:46.184594 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 8764
I20260927 21:27:46.184803 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/wal/instance
I20260927 21:27:46.185527 388 external_mini_cluster.cc:1428] Setting key e7d916228e71baec9fd2eea33c1746d8
Sep 27 21:27:46 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:27:46.201206 8894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:46.181429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33953 (local address 127.0.97.62:41753)
0927 21:27:46.182078 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:46.182088 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:46.183508 (+ 1420us) server_negotiation.cc:407] Connection header received
0927 21:27:46.186014 (+ 2506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:46.186020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:46.186112 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:46.186247 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:46.187858 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:46.188584 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:46.190358 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:46.190770 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:46.190923 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:46.194911 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:46.194917 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:46.194920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:46.194955 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:46.197753 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:46.198766 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:46.198773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:46.198779 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:46.198908 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:46.199414 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:46.199418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:46.199420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:46.199738 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:46.199847 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:46.200795 (+ 948us) server_negotiation.cc:299] Negotiation successful
0927 21:27:46.200967 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":325,"threads_started":1}
I20260927 21:27:46.202885 8895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:46.182174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.1:33953)
0927 21:27:46.183257 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:46.183368 (+ 111us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:46.185719 (+ 2351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:46.186549 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:46.186558 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:46.187091 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:46.187684 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:46.187702 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:46.188769 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:46.188774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:46.190191 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:46.190204 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:46.191036 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:46.191043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:46.191290 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:46.192011 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:46.192039 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:46.194642 (+ 2603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:46.197905 (+ 3263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:46.197918 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:46.197941 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:46.198595 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:46.199028 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:46.199035 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:46.199042 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:46.199296 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:46.199876 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:46.199887 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:46.200367 (+ 480us) client_negotiation.cc:769] Sending connection context
0927 21:27:46.200675 (+ 308us) client_negotiation.cc:240] Negotiation successful
0927 21:27:46.201296 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":434,"threads_started":1}
I20260927 21:27:46.204599 8892 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41753
I20260927 21:27:46.204929 8892 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:46.205760 8892 heartbeater.cc:511] Master 127.0.97.62:41753 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 21:27:46.208009 8703 ts_manager.cc:194] Registered new tserver with Master: 5c1182f9024c4cf19e2e2457d5cef552 (127.0.97.1:44145)
I20260927 21:27:46.209497 8703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 21:27:46.236564 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41753
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:46.567025 8898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:46.567317 8898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:46.567414 8898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:46.574708 8898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:46.574837 8898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:46.574952 8898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:27:46.584770 8898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:46.586063 8898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:46.587426 8898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:46.600656 8903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:46.600734 8904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:46.600711 8906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:46.606977 8898 server_base.cc:1034] running on GCE node
I20260927 21:27:46.607664 8898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:46.608628 8898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:46.609836 8898 hybrid_clock.cc:648] HybridClock initialized: now 1790544466609764 us; error 55 us; skew 500 ppm
I20260927 21:27:46.610185 8898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:46 dist-test-slave-bxm0 krb5kdc[8658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:46.613363 8898 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 21:27:46.615056 8898 webserver.cc:460] Webserver started at http://127.0.97.2:43985/ using document root <none> and password file <none>
I20260927 21:27:46.615852 8898 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:46.615995 8898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:46.616313 8898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:46.619344 8898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/data/instance:
uuid: "104d4e36b2a0488b8bee76a1e75fe146"
format_stamp: "Formatted at 2026-09-27 21:27:46 on dist-test-slave-bxm0"
server_key: "040eb8ed8835fa908b4cc8d2fa0bfe45"
server_key_iv: "8c7acedb273488c9c8e218a797374391"
server_key_version: "encryptionkey@0"
I20260927 21:27:46.620121 8898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/wal/instance:
uuid: "104d4e36b2a0488b8bee76a1e75fe146"
format_stamp: "Formatted at 2026-09-27 21:27:46 on dist-test-slave-bxm0"
server_key: "040eb8ed8835fa908b4cc8d2fa0bfe45"
server_key_iv: "8c7acedb273488c9c8e218a797374391"
server_key_version: "encryptionkey@0"
I20260927 21:27:46.625587 8898 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260927 21:27:46.629189 8913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:46.630607 8898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 21:27:46.630789 8898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "104d4e36b2a0488b8bee76a1e75fe146"
format_stamp: "Formatted at 2026-09-27 21:27:46 on dist-test-slave-bxm0"
server_key: "040eb8ed8835fa908b4cc8d2fa0bfe45"
server_key_iv: "8c7acedb273488c9c8e218a797374391"
server_key_version: "encryptionkey@0"
I20260927 21:27:46.631001 8898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:46.644840 8898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:46.649050 8898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:46.649348 8898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:46.650359 8898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:27:46.651672 8898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:46.651777 8898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:46.651929 8898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:46.652015 8898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:46.685410 8898 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33731
I20260927 21:27:46.685453 9025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33731 every 8 connection(s)
I20260927 21:27:46.686761 8898 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:27:46.689266 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 8898
I20260927 21:27:46.689474 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/wal/instance
I20260927 21:27:46.689975 388 external_mini_cluster.cc:1428] Setting key 2e2492c7a21fd0baa166e2f8d021d46f
Sep 27 21:27:46 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:27:46.710067 8894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:46.689499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48189 (local address 127.0.97.62:41753)
0927 21:27:46.689743 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:46.689749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:46.691171 (+ 1422us) server_negotiation.cc:407] Connection header received
0927 21:27:46.693753 (+ 2582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:46.693757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:46.693832 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:46.693949 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:46.696075 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:46.696763 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:46.699399 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:46.699980 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:46.700137 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:46.703740 (+ 3603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:46.703745 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:46.703748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:46.703788 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:46.706810 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:46.707575 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:46.707580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:46.707586 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:46.707710 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:46.708211 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:46.708216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:46.708221 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:46.708495 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:46.708650 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:46.709462 (+ 812us) server_negotiation.cc:299] Negotiation successful
0927 21:27:46.709715 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260927 21:27:46.711058 9028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:46.690104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.2:48189)
0927 21:27:46.690987 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:46.691045 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:46.693420 (+ 2375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:46.694205 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:46.694220 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:46.695056 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:46.695887 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:46.695914 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:46.696940 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:46.696948 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:46.699219 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:46.699237 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:46.700213 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:46.700218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:46.700446 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:46.701181 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:46.701206 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:46.703494 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:46.706969 (+ 3475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:46.706978 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:46.706997 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:46.707438 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:46.707879 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:46.707884 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:46.707888 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:46.708085 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:46.708626 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:46.708635 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:46.709093 (+ 458us) client_negotiation.cc:769] Sending connection context
0927 21:27:46.709363 (+ 270us) client_negotiation.cc:240] Negotiation successful
0927 21:27:46.709931 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":340,"threads_started":1}
I20260927 21:27:46.712818 9026 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41753
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 21:27:46.713183 9026 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:46.714075 9026 heartbeater.cc:511] Master 127.0.97.62:41753 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 21:27:46.715736 8703 ts_manager.cc:194] Registered new tserver with Master: 104d4e36b2a0488b8bee76a1e75fe146 (127.0.97.2:33731)
I20260927 21:27:46.716590 8703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 21:27:46.742743 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41753
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:47.072665 9031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:47.072984 9031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:47.073130 9031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:47.080524 9031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:47.080657 9031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:47.080856 9031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:27:47.090682 9031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:47.092028 9031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:47.093379 9031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:47.107244 9039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:47.107272 9037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:47.107340 9036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:47.108204 9031 server_base.cc:1034] running on GCE node
I20260927 21:27:47.108873 9031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:47.109979 9031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:47.111217 9031 hybrid_clock.cc:648] HybridClock initialized: now 1790544467111173 us; error 56 us; skew 500 ppm
I20260927 21:27:47.111578 9031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:47 dist-test-slave-bxm0 krb5kdc[8658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:47.114943 9031 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 21:27:47.116518 9031 webserver.cc:460] Webserver started at http://127.0.97.3:34889/ using document root <none> and password file <none>
I20260927 21:27:47.117246 9031 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:47.117426 9031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:47.117843 9031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:27:47.120849 9031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/data/instance:
uuid: "206bc670180a4908a5138463f7499917"
format_stamp: "Formatted at 2026-09-27 21:27:47 on dist-test-slave-bxm0"
server_key: "a453ed78cb62f2fc73c0901f74114907"
server_key_iv: "48259ef6de5dfd73a116c3fbc2e74e8f"
server_key_version: "encryptionkey@0"
I20260927 21:27:47.121753 9031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/wal/instance:
uuid: "206bc670180a4908a5138463f7499917"
format_stamp: "Formatted at 2026-09-27 21:27:47 on dist-test-slave-bxm0"
server_key: "a453ed78cb62f2fc73c0901f74114907"
server_key_iv: "48259ef6de5dfd73a116c3fbc2e74e8f"
server_key_version: "encryptionkey@0"
I20260927 21:27:47.127077 9031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 21:27:47.130611 9046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:47.132124 9031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:27:47.132342 9031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "206bc670180a4908a5138463f7499917"
format_stamp: "Formatted at 2026-09-27 21:27:47 on dist-test-slave-bxm0"
server_key: "a453ed78cb62f2fc73c0901f74114907"
server_key_iv: "48259ef6de5dfd73a116c3fbc2e74e8f"
server_key_version: "encryptionkey@0"
I20260927 21:27:47.132586 9031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:47.163470 9031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:47.168097 9031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:47.168418 9031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:47.169438 9031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:27:47.170856 9031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:47.170964 9031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:47.171118 9031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:47.171202 9031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:47.210124 9031 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39943
I20260927 21:27:47.210181 9158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39943 every 8 connection(s)
I20260927 21:27:47.211480 9031 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:27:47.212874 8892 heartbeater.cc:503] Master 127.0.97.62:41753 was elected leader, sending a full tablet report...
I20260927 21:27:47.216491 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 9031
I20260927 21:27:47.216663 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/wal/instance
I20260927 21:27:47.217098 388 external_mini_cluster.cc:1428] Setting key 8e79c752e148d8d659eaba355e3b632d
Sep 27 21:27:47 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:27:47.234040 9161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:47.214244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49031 (local address 127.0.97.62:41753)
0927 21:27:47.218027 (+ 3783us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:47.218033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:47.218054 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:27:47.219100 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:47.219105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:47.219196 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:47.219287 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:47.221088 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:47.222008 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:47.223779 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:47.224178 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:47.224286 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:47.227910 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:47.227915 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:47.227918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:47.227953 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:47.230556 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:47.231512 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:47.231517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:47.231520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:47.231677 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:47.232212 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:47.232217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:47.232223 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:47.232632 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:47.232819 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:47.233511 (+ 692us) server_negotiation.cc:299] Negotiation successful
0927 21:27:47.233763 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3658,"thread_start_us":3537,"threads_started":1}
I20260927 21:27:47.235105 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:47.214979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.3:49031)
0927 21:27:47.215978 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:47.216048 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:47.218767 (+ 2719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:47.219588 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:47.219600 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:47.220349 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:47.220935 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:47.220953 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:47.222156 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:47.222159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:47.223589 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:47.223600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:47.224415 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:47.224421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:47.224636 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:47.225322 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:47.225344 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:47.227647 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:27:47.230716 (+ 3069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:47.230738 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:47.230764 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:47.231384 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:47.231804 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:27:47.231809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:27:47.231813 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:27:47.232082 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:27:47.232778 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:27:47.232787 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:27:47.233159 (+ 372us) client_negotiation.cc:769] Sending connection context
0927 21:27:47.233401 (+ 242us) client_negotiation.cc:240] Negotiation successful
0927 21:27:47.233892 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":380,"threads_started":1}
I20260927 21:27:47.236445 9159 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41753
I20260927 21:27:47.236744 9159 heartbeater.cc:464] Registering TS with master...
I20260927 21:27:47.237526 9159 heartbeater.cc:511] Master 127.0.97.62:41753 requested a full tablet report, sending...
I20260927 21:27:47.238989 8703 ts_manager.cc:194] Registered new tserver with Master: 206bc670180a4908a5138463f7499917 (127.0.97.3:39943)
I20260927 21:27:47.239842 8703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49031
I20260927 21:27:47.244947 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260927 21:27:47.385022 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 8764
Sep 27 21:27:47 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 21:27:47.401739 9170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:47.386312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47692 (local address 127.0.97.1:44145)
0927 21:27:47.386967 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:47.386975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:47.387002 (+ 27us) server_negotiation.cc:407] Connection header received
0927 21:27:47.387061 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:47.387068 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:47.387279 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:47.387411 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:47.388714 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:47.389877 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:47.391096 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:47.391653 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:47.391767 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:47.395984 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:47.395996 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:47.396011 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:47.396046 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:47.398911 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:47.399576 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:47.399586 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:47.399596 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:47.399719 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:47.400112 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:47.400122 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:47.400124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:47.400463 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:47.400775 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:47.401065 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 21:27:47.401328 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":368,"threads_started":1}
I20260927 21:27:47.402999 8887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:47.486188 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 8764
I20260927 21:27:47.502813 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 8898
Sep 27 21:27:47 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 21:27:47.520022 9172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:47.503813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34702 (local address 127.0.97.2:33731)
0927 21:27:47.504322 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:47.504329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:47.504356 (+ 27us) server_negotiation.cc:407] Connection header received
0927 21:27:47.504420 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:47.504427 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:47.504651 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:47.504852 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:47.506327 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:47.507373 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:47.508780 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:47.509365 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:47.509525 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:47.514170 (+ 4645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:47.514183 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:47.514198 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:47.514236 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:47.517135 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:47.517959 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:47.517966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:47.517968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:47.518089 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:47.518623 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:47.518626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:47.518628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:47.518979 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:47.519207 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:47.519501 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 21:27:47.519705 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":258,"threads_started":1}
I20260927 21:27:47.521405 9021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:47.603173 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 8898
I20260927 21:27:47.619181 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 9031
Sep 27 21:27:47 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 21:27:47.639979 9174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:47.620435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41910 (local address 127.0.97.3:39943)
0927 21:27:47.621130 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:47.621137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:47.621167 (+ 30us) server_negotiation.cc:407] Connection header received
0927 21:27:47.621235 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:47.621242 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:47.621472 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:47.621603 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:47.623718 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:47.625721 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:47.627637 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:47.628722 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:47.628879 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:47.634012 (+ 5133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:47.634025 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:47.634041 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:47.634082 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:47.636932 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:47.637815 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:47.637823 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:47.637826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:47.637985 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:47.638338 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:47.638342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:47.638344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:47.638744 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:47.639024 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:47.639312 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 21:27:47.639580 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":341,"threads_started":1}
I20260927 21:27:47.641306 9154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:47.725667 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 9031
I20260927 21:27:47.741823 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 8671
I20260927 21:27:47.743064 8733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:27:47.845384 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 8671
I20260927 21:27:47.866995 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41753
--webserver_interface=127.0.97.62
--webserver_port=41659
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:27:48.206902 9177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:48.207203 9177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:48.207295 9177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:48.214713 9177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:27:48.214843 9177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:48.214890 9177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:48.214949 9177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:27:48.214998 9177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:27:48.225154 9177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=41659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:48.226585 9177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:48.227911 9177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:48.240147 9185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:48.240262 9182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:48.240276 9183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:48.240998 9177 server_base.cc:1034] running on GCE node
I20260927 21:27:48.241909 9177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:48.243386 9177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:48.244596 9177 hybrid_clock.cc:648] HybridClock initialized: now 1790544468244550 us; error 44 us; skew 500 ppm
I20260927 21:27:48.244966 9177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:48 dist-test-slave-bxm0 krb5kdc[8658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:48.248154 9177 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:27:48.249888 9177 webserver.cc:460] Webserver started at http://127.0.97.62:41659/ using document root <none> and password file <none>
I20260927 21:27:48.250636 9177 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:48.250782 9177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:48.256356 9177 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:27:48.259369 9192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:48.260897 9177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 21:27:48.261089 9177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "ff4e61de07784def89452725e0f3bb2d"
format_stamp: "Formatted at 2026-09-27 21:27:45 on dist-test-slave-bxm0"
server_key: "4e5f4ef65c4d548e05bd86aef52bf8ee"
server_key_iv: "5a844bebf420b16aa5a116c7cab11819"
server_key_version: "encryptionkey@0"
I20260927 21:27:48.262162 9177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:48.279160 9177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:48.283929 9177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:48.284256 9177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:48.303006 9177 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41753
I20260927 21:27:48.303042 9243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41753 every 8 connection(s)
I20260927 21:27:48.304519 9177 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:27:48.309979 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 9177
I20260927 21:27:48.310731 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:44145
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41753
--webserver_port=37047
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:27:48.311604 9244 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:27:48.314941 9244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d: Bootstrap starting.
I20260927 21:27:48.333838 9244 log.cc:824] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d: Log is configured to *not* fsync() on all Append() calls
I20260927 21:27:48.342650 9244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:27:48.343225 9244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d: Bootstrap complete.
I20260927 21:27:48.352397 9244 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4e61de07784def89452725e0f3bb2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41753 } }
I20260927 21:27:48.353235 9244 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff4e61de07784def89452725e0f3bb2d, State: Initialized, Role: FOLLOWER
I20260927 21:27:48.353974 9244 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4e61de07784def89452725e0f3bb2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41753 } }
I20260927 21:27:48.354206 9244 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:27:48.354323 9244 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:27:48.354516 9244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:27:48.358877 9244 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4e61de07784def89452725e0f3bb2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41753 } }
I20260927 21:27:48.359412 9244 leader_election.cc:302] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff4e61de07784def89452725e0f3bb2d; no voters:
I20260927 21:27:48.360055 9244 leader_election.cc:288] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 21:27:48.360186 9248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:27:48.361397 9248 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [term 2 LEADER]: Becoming Leader. State: Replica: ff4e61de07784def89452725e0f3bb2d, State: Running, Role: LEADER
I20260927 21:27:48.362068 9248 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4e61de07784def89452725e0f3bb2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41753 } }
I20260927 21:27:48.362382 9244 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:27:48.364748 9250 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff4e61de07784def89452725e0f3bb2d. Latest consensus state: current_term: 2 leader_uuid: "ff4e61de07784def89452725e0f3bb2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4e61de07784def89452725e0f3bb2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41753 } } }
I20260927 21:27:48.365115 9250 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:48.364826 9249 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ff4e61de07784def89452725e0f3bb2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff4e61de07784def89452725e0f3bb2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41753 } } }
I20260927 21:27:48.365231 9249 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d [sys.catalog]: This master's current role is: LEADER
I20260927 21:27:48.376300 9253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:27:48.385777 9253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:27:48.387389 9253 catalog_manager.cc:1259] Loaded cluster ID: 867765f7cb3b487ea36e3a52d2ba7775
I20260927 21:27:48.387598 9253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:27:48.391503 9253 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:27:48.393074 9253 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ff4e61de07784def89452725e0f3bb2d: Loaded TSK: 0
I20260927 21:27:48.394761 9253 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:27:48.677867 9246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:48.678258 9246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:48.678453 9246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:48.690776 9246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:48.690925 9246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:48.691107 9246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:27:48.705931 9246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:44145
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=37047
--tserver_master_addrs=127.0.97.62:41753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:48.707377 9246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:48.708762 9246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:48.722397 9271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:48.723050 9273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:48.723300 9270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:48.723636 9246 server_base.cc:1034] running on GCE node
I20260927 21:27:48.724251 9246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:48.725241 9246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:48.726500 9246 hybrid_clock.cc:648] HybridClock initialized: now 1790544468726442 us; error 48 us; skew 500 ppm
I20260927 21:27:48.726886 9246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:48 dist-test-slave-bxm0 krb5kdc[8658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544468, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:48.730199 9246 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 21:27:48.731818 9246 webserver.cc:460] Webserver started at http://127.0.97.1:37047/ using document root <none> and password file <none>
I20260927 21:27:48.732525 9246 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:48.732681 9246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:48.738250 9246 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 21:27:48.741053 9280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:48.742554 9246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:27:48.742733 9246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "5c1182f9024c4cf19e2e2457d5cef552"
format_stamp: "Formatted at 2026-09-27 21:27:46 on dist-test-slave-bxm0"
server_key: "cdf33c08a45b90c6b5f8c489163d6cf2"
server_key_iv: "71471d3b35a1aa23b70d77dd8c32cd00"
server_key_version: "encryptionkey@0"
I20260927 21:27:48.743723 9246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:48.757172 9246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:48.761988 9246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:48.762324 9246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:48.763286 9246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:27:48.764642 9246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:48.764767 9246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:48.764902 9246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:48.764988 9246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:48.798911 9246 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44145
I20260927 21:27:48.798962 9392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44145 every 8 connection(s)
I20260927 21:27:48.800372 9246 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:27:48.803361 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 9246
I20260927 21:27:48.804102 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33731
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41753
--webserver_port=43985
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:27:48 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:48 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:48.820591 9396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:48.803686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.1:59935)
0927 21:27:48.805239 (+ 1553us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:48.805310 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:48.807234 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:48.808879 (+ 1645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:48.808888 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:48.809397 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:48.810038 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:48.810057 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:48.812159 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:48.812162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:48.813449 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:48.813461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:48.814984 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:48.814988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:48.815189 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:48.815920 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:48.815943 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:48.819585 (+ 3642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1140,"thread_start_us":876,"threads_started":1}
W20260927 21:27:48.821333 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:48.803417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59935 (local address 127.0.97.62:41753)
0927 21:27:48.806136 (+ 2719us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:48.806155 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:48.806210 (+ 55us) server_negotiation.cc:407] Connection header received
0927 21:27:48.807873 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:48.807894 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:48.808351 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:48.808777 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:48.810243 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:48.811980 (+ 1737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:48.813602 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:48.814773 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:48.814961 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:48.820058 (+ 5097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59935: BlockingRecv error: recv got EOF from 127.0.97.1:59935 (error 108)
Metrics: {"server-negotiator.queue_time_us":2440,"thread_start_us":2202,"threads_started":1}
W20260927 21:27:48.821677 9393 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41753 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:41753: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 27 21:27:48 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:48 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:48.831892 9396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:48.822351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.1:44351)
0927 21:27:48.822641 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:48.822664 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:48.822796 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:48.823153 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:48.823160 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:48.823642 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:48.823881 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:48.823898 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:48.824908 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:48.824916 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:48.826665 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:48.826683 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:48.827362 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:48.827369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:48.827591 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:48.828233 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:48.828263 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:48.831594 (+ 3331us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141}
W20260927 21:27:48.832176 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:48.822512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44351 (local address 127.0.97.62:41753)
0927 21:27:48.822682 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:48.822687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:48.822721 (+ 34us) server_negotiation.cc:407] Connection header received
0927 21:27:48.822935 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:48.822938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:48.822998 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:48.823075 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:48.824034 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:48.824755 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:48.826816 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:48.827207 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:48.827301 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:48.831796 (+ 4495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:44351: BlockingRecv error: recv got EOF from 127.0.97.1:44351 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 21:27:48 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:48 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:48.840546 9396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:48.832604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.1:50275)
0927 21:27:48.832776 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:48.832795 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:48.832897 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:48.833225 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:48.833230 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:48.833678 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:48.833836 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:48.833845 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:48.834816 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:48.834822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:48.835979 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:48.835991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:48.836664 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:48.836670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:48.836831 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:48.837436 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:48.837454 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:48.840267 (+ 2813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 21:27:48.840695 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:48.832697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50275 (local address 127.0.97.62:41753)
0927 21:27:48.832933 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:48.832941 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:48.832961 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:27:48.833012 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:48.833017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:48.833100 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:48.833201 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:48.833972 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:48.834679 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:48.836125 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:48.836519 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:48.836642 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:48.840447 (+ 3805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50275: BlockingRecv error: recv got EOF from 127.0.97.1:50275 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":713,"threads_started":1}
W20260927 21:27:48.840862 9393 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:27:49.139605 9397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:49.139954 9397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:49.140129 9397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:49.147624 9397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:49.147773 9397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:49.147992 9397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:27:49.157972 9397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43985
--tserver_master_addrs=127.0.97.62:41753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:49.159346 9397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:49.160629 9397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:49.173772 9406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:49.173833 9404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:49.173739 9403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:49.174906 9397 server_base.cc:1034] running on GCE node
I20260927 21:27:49.175585 9397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:49.176625 9397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:49.177891 9397 hybrid_clock.cc:648] HybridClock initialized: now 1790544469177874 us; error 93 us; skew 500 ppm
I20260927 21:27:49.178167 9397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:49 dist-test-slave-bxm0 krb5kdc[8658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544469, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:49.181253 9397 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 21:27:49.182636 9397 webserver.cc:460] Webserver started at http://127.0.97.2:43985/ using document root <none> and password file <none>
I20260927 21:27:49.183290 9397 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:49.183398 9397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:49.188750 9397 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 21:27:49.191665 9413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:49.193506 9397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:27:49.193712 9397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "104d4e36b2a0488b8bee76a1e75fe146"
format_stamp: "Formatted at 2026-09-27 21:27:46 on dist-test-slave-bxm0"
server_key: "040eb8ed8835fa908b4cc8d2fa0bfe45"
server_key_iv: "8c7acedb273488c9c8e218a797374391"
server_key_version: "encryptionkey@0"
I20260927 21:27:49.195223 9397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:49.206749 9397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:49.211210 9397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:49.211510 9397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:49.212544 9397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:27:49.213961 9397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:49.214059 9397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:49.214210 9397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:49.214313 9397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:49.248445 9397 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33731
I20260927 21:27:49.248534 9525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33731 every 8 connection(s)
I20260927 21:27:49.249918 9397 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:27:49.259727 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 9397
I20260927 21:27:49.260504 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39943
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41753
--webserver_port=34889
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:27:49 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:49 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 21:27:49.267805 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:49.252482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33995 (local address 127.0.97.62:41753)
0927 21:27:49.252687 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:49.252694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:49.254004 (+ 1310us) server_negotiation.cc:407] Connection header received
0927 21:27:49.255890 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:49.255894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:49.255961 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:49.256053 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:49.257589 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.258439 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.260451 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.260953 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.261059 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:49.267501 (+ 6442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33995: BlockingRecv error: recv got EOF from 127.0.97.2:33995 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 21:27:49.268561 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:49.253011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.2:33995)
0927 21:27:49.253829 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:49.253878 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:49.255609 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:49.256301 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:49.256309 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:49.256824 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:49.257393 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:49.257410 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.258596 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.258599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:49.260294 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:49.260309 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.261186 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.261196 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:49.261549 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:49.262412 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:49.262448 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:49.267298 (+ 4850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":296,"threads_started":1}
W20260927 21:27:49.269296 9526 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41753 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:41753: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 27 21:27:49 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:49 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:49.279948 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:49.269856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.2:42803)
0927 21:27:49.270057 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:49.270078 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:49.270190 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:49.270487 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:49.270495 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:49.270955 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:49.271167 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:49.271181 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.272186 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.272194 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:49.273918 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:49.273936 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.274720 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.274728 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:49.274963 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:49.275588 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:49.275615 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:49.279539 (+ 3924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 21:27:49.280051 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:49.270050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42803 (local address 127.0.97.62:41753)
0927 21:27:49.270204 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:49.270208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:49.270224 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:27:49.270293 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:49.270296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:49.270353 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:49.270431 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:49.271336 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.272035 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.274106 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.274551 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.274671 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:49.279823 (+ 5152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:42803: BlockingRecv error: recv got EOF from 127.0.97.2:42803 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 21:27:49 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:49 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:49.288834 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:49.280976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.2:55927)
0927 21:27:49.281172 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:49.281192 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:49.281328 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:49.281696 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:49.281700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:49.282032 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:49.282181 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:49.282190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.283187 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.283194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:49.284134 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:49.284152 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.284845 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.284850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:49.285022 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:49.285700 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:49.285717 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:49.288510 (+ 2793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 21:27:49.288982 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:49.281090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55927 (local address 127.0.97.62:41753)
0927 21:27:49.281273 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:49.281289 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:49.281308 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:27:49.281488 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:49.281492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:49.281550 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:49.281673 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:49.282328 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.283045 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.284307 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.284691 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.284834 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:49.288744 (+ 3910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55927: BlockingRecv error: recv got EOF from 127.0.97.2:55927 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 21:27:49.289278 9526 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:27:49.600103 9529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:27:49.600394 9529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:27:49.600486 9529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:27:49.607846 9529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:27:49.607968 9529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:27:49.608083 9529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:27:49.618081 9529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34889
--tserver_master_addrs=127.0.97.62:41753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:27:49.619359 9529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:27:49.620612 9529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:27:49.636832 9534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:49.636852 9535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:27:49.637213 9537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:27:49.637931 9529 server_base.cc:1034] running on GCE node
I20260927 21:27:49.638558 9529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:27:49.639588 9529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:27:49.640815 9529 hybrid_clock.cc:648] HybridClock initialized: now 1790544469640765 us; error 45 us; skew 500 ppm
I20260927 21:27:49.641146 9529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:27:49 dist-test-slave-bxm0 krb5kdc[8658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544469, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:27:49.644232 9529 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 21:27:49.645742 9529 webserver.cc:460] Webserver started at http://127.0.97.3:34889/ using document root <none> and password file <none>
I20260927 21:27:49.646512 9529 fs_manager.cc:359] Metadata directory not provided
I20260927 21:27:49.646670 9529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:27:49.652406 9529 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:27:49.655531 9544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:49.656867 9529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:27:49.657056 9529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "206bc670180a4908a5138463f7499917"
format_stamp: "Formatted at 2026-09-27 21:27:47 on dist-test-slave-bxm0"
server_key: "a453ed78cb62f2fc73c0901f74114907"
server_key_iv: "48259ef6de5dfd73a116c3fbc2e74e8f"
server_key_version: "encryptionkey@0"
I20260927 21:27:49.658090 9529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:27:49.677978 9529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:27:49.682044 9529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:27:49.682369 9529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:27:49.683336 9529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:27:49.684636 9529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:27:49.684731 9529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:49.684813 9529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:27:49.684865 9529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:27:49.719679 9529 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39943
I20260927 21:27:49.719823 9656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39943 every 8 connection(s)
I20260927 21:27:49.721191 9529 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:27:49.727089 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 9529
Sep 27 21:27:49 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:49 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 21:27:49.737773 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:49.724004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47115 (local address 127.0.97.62:41753)
0927 21:27:49.724180 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:49.724190 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:49.725364 (+ 1174us) server_negotiation.cc:407] Connection header received
0927 21:27:49.727262 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:49.727267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:49.727362 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:49.727477 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:49.729276 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.730071 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.731641 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.732060 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.732191 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:49.737477 (+ 5286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47115: BlockingRecv error: recv got EOF from 127.0.97.3:47115 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 21:27:49.738541 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:49.724473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.3:47115)
0927 21:27:49.725188 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:49.725226 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:49.726983 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:49.727722 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:49.727731 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:49.728310 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:49.729108 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:49.729126 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.730239 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.730243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:49.731491 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:49.731503 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.732290 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.732297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:49.732552 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:49.733270 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:49.733298 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:49.737270 (+ 3972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":294,"threads_started":1}
W20260927 21:27:49.739259 9657 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41753 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:41753: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260927 21:27:49.743115 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:49.728288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48186 (local address 127.0.97.62:41753)
0927 21:27:49.728977 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:49.728984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:49.729006 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:27:49.729058 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:49.729063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:49.729127 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:49.729238 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:49.730782 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.731545 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.732774 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.733180 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.733304 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:49.736206 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:27:49.736214 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:27:49.736234 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:27:49.736270 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:27:49.739706 (+ 3436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:49.740553 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:49.740560 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:49.740569 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:49.740726 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:27:49.741139 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:27:49.741144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:27:49.741146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:27:49.741767 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:27:49.742101 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0927 21:27:49.742464 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 21:27:49.742826 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":474,"threads_started":1}
Sep 27 21:27:49 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:49 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:49.748533 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:49.739882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.3:53181)
0927 21:27:49.740054 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:49.740074 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:49.740171 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:49.740470 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:49.740477 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:49.740808 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:49.741012 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:49.741023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.742476 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.742482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:49.743608 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:49.743619 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.744346 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.744351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:49.744475 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:49.745053 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:49.745070 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:49.748233 (+ 3163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 21:27:49.748639 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:49.739955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53181 (local address 127.0.97.62:41753)
0927 21:27:49.740121 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:49.740125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:49.740139 (+ 14us) server_negotiation.cc:407] Connection header received
0927 21:27:49.740274 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:49.740278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:49.740330 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:49.740407 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:49.741140 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.742328 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.743750 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.744203 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.744340 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:49.748423 (+ 4083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53181: BlockingRecv error: recv got EOF from 127.0.97.3:53181 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 21:27:49 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:49 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:49.756470 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:49.749225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.3:56347)
0927 21:27:49.749432 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:49.749452 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:49.749548 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:49.749858 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:49.749862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:49.750160 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:49.750292 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:49.750300 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.751279 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.751286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:49.752276 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:49.752286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.753009 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.753015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:49.753173 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:49.753712 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:49.753728 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:49.756263 (+ 2535us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 21:27:49.756760 9657 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:27:49.756785 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:49.749326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56347 (local address 127.0.97.62:41753)
0927 21:27:49.749547 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:49.749553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:49.749569 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:27:49.749673 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:49.749678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:49.749744 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:49.749839 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:49.750431 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.751151 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.752415 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.752849 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.752988 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:49.756470 (+ 3482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56347: BlockingRecv error: recv got EOF from 127.0.97.3:56347 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
Sep 27 21:27:49 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:49 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:49.850837 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:49.841504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.1:56999)
0927 21:27:49.842198 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:49.842220 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:49.842402 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:49.842776 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:49.842781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:49.843307 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:49.843521 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:49.843531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.844893 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.844898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:49.846100 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:49.846112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.846836 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.846841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:49.847055 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:49.847689 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:49.847707 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:49.850578 (+ 2871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":428,"threads_started":1}
W20260927 21:27:49.851099 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:49.841662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56999 (local address 127.0.97.62:41753)
0927 21:27:49.841946 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:49.841956 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:49.842322 (+ 366us) server_negotiation.cc:407] Connection header received
0927 21:27:49.842545 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:49.842556 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:49.842621 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:49.842739 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:49.843698 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.844712 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.846245 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:49.846655 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:49.846763 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:49.850786 (+ 4023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56999: BlockingRecv error: recv got EOF from 127.0.97.1:56999 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 27 21:27:50 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:50 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:50.298512 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:50.289912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.2:37633)
0927 21:27:50.290449 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:50.290465 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:50.290575 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:50.290924 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:50.290928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:50.291395 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:50.291594 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:50.291602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:50.292577 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:50.292581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:50.293545 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:50.293557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:50.294356 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:50.294360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:50.294490 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:50.295162 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:50.295179 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:50.298251 (+ 3072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":311,"threads_started":1}
W20260927 21:27:50.298791 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:50.290074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37633 (local address 127.0.97.62:41753)
0927 21:27:50.290245 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:50.290251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:50.290565 (+ 314us) server_negotiation.cc:407] Connection header received
0927 21:27:50.290675 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:50.290679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:50.290742 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:50.290838 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:50.291733 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:50.292429 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:50.293778 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:50.294182 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:50.294295 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:50.298471 (+ 4176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37633: BlockingRecv error: recv got EOF from 127.0.97.2:37633 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 21:27:50 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:50 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:50.770867 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:50.757494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.3:48829)
0927 21:27:50.758089 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:50.758108 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:50.758238 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:50.758556 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:50.758560 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:50.759064 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:50.759276 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:50.759285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:50.760515 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:50.760522 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:50.761732 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:50.761745 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:50.762662 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:50.762669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:50.762874 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:50.767367 (+ 4493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:50.767399 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:50.770512 (+ 3113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":339,"threads_started":1}
W20260927 21:27:50.770969 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:50.757661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48829 (local address 127.0.97.62:41753)
0927 21:27:50.757861 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:50.757865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:50.758199 (+ 334us) server_negotiation.cc:407] Connection header received
0927 21:27:50.758345 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:50.758349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:50.758413 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:50.758504 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:50.759448 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:50.760345 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:50.761882 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:50.762473 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:50.762628 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:50.770749 (+ 8121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:48829: BlockingRecv error: recv got EOF from 127.0.97.3:48829 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 21:27:50 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:50 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:50.860949 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:50.851657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.1:48731)
0927 21:27:50.852174 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:50.852190 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:50.852313 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:50.852633 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:50.852637 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:50.853016 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:50.853153 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:50.853161 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:50.854296 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:50.854300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:50.855453 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:50.855469 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:50.856433 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:50.856438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:50.856548 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:50.857134 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:50.857152 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:50.860573 (+ 3421us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":322,"threads_started":1}
W20260927 21:27:50.861006 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:50.851761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48731 (local address 127.0.97.62:41753)
0927 21:27:50.852023 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:50.852029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:50.852308 (+ 279us) server_negotiation.cc:407] Connection header received
0927 21:27:50.852400 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:50.852404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:50.852494 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:50.852582 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:50.853357 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:50.854151 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:50.855662 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:50.856275 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:50.856393 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:50.860760 (+ 4367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48731: BlockingRecv error: recv got EOF from 127.0.97.1:48731 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 21:27:51 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:51 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:51.308528 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:51.299593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.2:55983)
0927 21:27:51.300088 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:51.300104 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:51.300230 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:51.300553 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:51.300558 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:51.301007 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:51.301168 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:51.301178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:51.302257 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:51.302263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:51.303275 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:51.303288 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:51.304045 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:51.304051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:51.304215 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:51.304877 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:51.304902 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:51.308173 (+ 3271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":301,"threads_started":1}
W20260927 21:27:51.308597 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:51.299686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55983 (local address 127.0.97.62:41753)
0927 21:27:51.299955 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:51.299961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:51.300199 (+ 238us) server_negotiation.cc:407] Connection header received
0927 21:27:51.300344 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:51.300348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:51.300410 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:51.300528 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:51.301321 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:51.302117 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:51.303420 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:51.303885 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:51.304054 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:51.308372 (+ 4318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55983: BlockingRecv error: recv got EOF from 127.0.97.2:55983 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 27 21:27:51 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:51 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:51.780972 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:51.771879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.3:34123)
0927 21:27:51.772390 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:51.772406 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:51.772538 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:51.773162 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:51.773168 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:51.773727 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:51.773879 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:51.773888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:51.775293 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:51.775297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:51.776222 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:51.776234 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:51.776875 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:51.776879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:51.776992 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:51.777553 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:51.777569 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:51.780690 (+ 3121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":263,"threads_started":1}
W20260927 21:27:51.781224 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:51.772158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34123 (local address 127.0.97.62:41753)
0927 21:27:51.772409 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:51.772414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:51.772502 (+ 88us) server_negotiation.cc:407] Connection header received
0927 21:27:51.772706 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:51.772712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:51.772792 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:51.772925 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:51.774045 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:51.775132 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:51.776365 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:51.776748 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:51.776839 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:51.780929 (+ 4090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34123: BlockingRecv error: recv got EOF from 127.0.97.3:34123 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 27 21:27:51 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:51 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:51.870462 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:51.861912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.1:57747)
0927 21:27:51.862490 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:51.862506 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:51.862656 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:51.862988 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:51.862993 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:51.863438 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:51.863599 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:51.863607 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:51.864659 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:51.864667 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:51.865588 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:51.865600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:51.866414 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:51.866420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:51.866589 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:51.867246 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:51.867269 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:51.870212 (+ 2943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":332,"threads_started":1}
W20260927 21:27:51.870750 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:51.862065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57747 (local address 127.0.97.62:41753)
0927 21:27:51.862365 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:51.862372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:51.862586 (+ 214us) server_negotiation.cc:407] Connection header received
0927 21:27:51.862776 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:51.862782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:51.862857 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:51.862958 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:51.863793 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:51.864520 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:51.865764 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:51.866256 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:51.866395 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:51.870442 (+ 4047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57747: BlockingRecv error: recv got EOF from 127.0.97.1:57747 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 27 21:27:52 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:52 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:52.317512 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:52.309694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.2:32791)
0927 21:27:52.310126 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:52.310141 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:52.310257 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:52.310558 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:52.310561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:52.310884 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:52.311051 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:52.311059 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:52.312089 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:52.312092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:52.312985 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:52.312996 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:52.313693 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:52.313698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:52.313918 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:52.314539 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:52.314555 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:52.317255 (+ 2700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":218,"threads_started":1}
W20260927 21:27:52.317781 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:52.309693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32791 (local address 127.0.97.62:41753)
0927 21:27:52.309902 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:52.309912 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:52.310238 (+ 326us) server_negotiation.cc:407] Connection header received
0927 21:27:52.310370 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:52.310374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:52.310437 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:52.310522 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:52.311195 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:52.311965 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:52.313103 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:52.313517 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:52.313710 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:52.317482 (+ 3772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:32791: BlockingRecv error: recv got EOF from 127.0.97.2:32791 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 21:27:52 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:52 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:52.790781 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:52.782139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.3:43855)
0927 21:27:52.782713 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:52.782730 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:52.782868 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:52.783230 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:52.783234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:52.783586 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:52.783759 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:52.783767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:52.784919 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:52.784923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:52.785856 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:52.785869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:52.786587 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:52.786591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:52.786714 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:52.787334 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:52.787354 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:52.790540 (+ 3186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":315,"threads_started":1}
W20260927 21:27:52.791031 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:52.782232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43855 (local address 127.0.97.62:41753)
0927 21:27:52.782482 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:52.782492 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:52.782856 (+ 364us) server_negotiation.cc:407] Connection header received
0927 21:27:52.782981 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:52.782984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:52.783066 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:52.783155 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:52.783937 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:52.784671 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:52.786017 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:52.786451 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:52.786559 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:52.790723 (+ 4164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43855: BlockingRecv error: recv got EOF from 127.0.97.3:43855 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 27 21:27:52 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:52 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:52.879925 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:52.871524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.1:56001)
0927 21:27:52.872025 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:52.872042 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:52.872162 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:52.872553 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:52.872556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:52.872938 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:52.873081 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:52.873090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:52.874118 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:52.874121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:52.875061 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:52.875072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:52.875929 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:52.875935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:52.876118 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:52.876760 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:52.876778 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:52.879587 (+ 2809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":279,"threads_started":1}
W20260927 21:27:52.880020 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:52.871692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56001 (local address 127.0.97.62:41753)
0927 21:27:52.871948 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:52.871955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:52.872127 (+ 172us) server_negotiation.cc:407] Connection header received
0927 21:27:52.872320 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:52.872325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:52.872392 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:52.872503 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:52.873250 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:52.874002 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:52.875212 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:52.875757 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:52.875908 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:52.879789 (+ 3881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56001: BlockingRecv error: recv got EOF from 127.0.97.1:56001 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Sep 27 21:27:53 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:53 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:53.327006 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:53.318587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.2:44895)
0927 21:27:53.319119 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:53.319135 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:53.319298 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:53.319641 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:53.319646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:53.320034 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:53.320189 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:53.320197 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:53.321185 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:53.321190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:53.322259 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:53.322271 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:53.322976 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:53.322981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:53.323167 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:53.323823 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:53.323840 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:53.326643 (+ 2803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":296,"threads_started":1}
W20260927 21:27:53.327080 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:53.318686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44895 (local address 127.0.97.62:41753)
0927 21:27:53.318942 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:53.318950 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:53.319224 (+ 274us) server_negotiation.cc:407] Connection header received
0927 21:27:53.319442 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:53.319446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:53.319507 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:53.319612 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:53.320351 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:53.321043 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:53.322406 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:53.322828 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:53.322964 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:53.326847 (+ 3883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44895: BlockingRecv error: recv got EOF from 127.0.97.2:44895 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 27 21:27:53 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:53 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:53.801007 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:53.791782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.3:37387)
0927 21:27:53.792338 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:53.792354 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:53.792487 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:53.792913 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:53.792917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:53.793265 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:53.793407 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:53.793414 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:53.794609 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:53.794639 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:53.795907 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:53.795919 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:53.796705 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:53.796710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:53.796906 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:53.797573 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:53.797594 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:53.800661 (+ 3067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":299,"threads_started":1}
W20260927 21:27:53.801116 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:53.791901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37387 (local address 127.0.97.62:41753)
0927 21:27:53.792073 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:53.792078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:53.792462 (+ 384us) server_negotiation.cc:407] Connection header received
0927 21:27:53.792597 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:53.792601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:53.792666 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:53.792778 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:53.793547 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:53.794457 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:53.796096 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:53.796552 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:53.796648 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:53.800870 (+ 4222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37387: BlockingRecv error: recv got EOF from 127.0.97.3:37387 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 21:27:53 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:53 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:53.890151 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:53.881054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.1:55547)
0927 21:27:53.881756 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:53.881778 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:53.881937 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:53.882301 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:53.882304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:53.882862 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:53.883050 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:53.883059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:53.884077 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:53.884081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:53.885068 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:53.885079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:53.885882 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:53.885891 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:53.886127 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:53.886734 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:53.886753 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:53.889789 (+ 3036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":360,"threads_started":1}
W20260927 21:27:53.890264 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:53.881239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55547 (local address 127.0.97.62:41753)
0927 21:27:53.881474 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:53.881485 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:53.881871 (+ 386us) server_negotiation.cc:407] Connection header received
0927 21:27:53.882053 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:53.882057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:53.882119 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:53.882247 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:53.883188 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:53.883882 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:53.885209 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:53.885716 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:53.885831 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:53.890044 (+ 4213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55547: BlockingRecv error: recv got EOF from 127.0.97.1:55547 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 27 21:27:54 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:54 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:54.336954 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:54.328201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.2:39367)
0927 21:27:54.328860 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:54.328878 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:54.329060 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:54.329431 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:54.329437 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:54.329894 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:54.330070 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:54.330080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:54.331147 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:54.331156 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:54.332130 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:54.332142 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:54.332905 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:54.332911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:54.333103 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:54.333732 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:54.333752 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:54.336584 (+ 2832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":358,"threads_started":1}
W20260927 21:27:54.337062 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:54.328357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39367 (local address 127.0.97.62:41753)
0927 21:27:54.328656 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:54.328662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:54.328988 (+ 326us) server_negotiation.cc:407] Connection header received
0927 21:27:54.329195 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:54.329204 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:54.329283 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:54.329400 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:54.330232 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:54.330995 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:54.332293 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:54.332749 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:54.332895 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:54.336806 (+ 3911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39367: BlockingRecv error: recv got EOF from 127.0.97.2:39367 (error 108)
Metrics: {"server-negotiator.queue_time_us":167}
Sep 27 21:27:54 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:54 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:54.810590 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:54.802094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.3:55187)
0927 21:27:54.802553 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:54.802569 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:54.802680 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:54.803000 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:54.803003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:54.803392 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:54.803549 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:54.803557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:54.804577 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:54.804581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:54.805538 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:54.805550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:54.806323 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:54.806327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:54.806444 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:54.807102 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:54.807119 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:54.810250 (+ 3131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":211,"threads_started":1}
W20260927 21:27:54.810702 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:54.802256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55187 (local address 127.0.97.62:41753)
0927 21:27:54.802464 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:54.802473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:54.802645 (+ 172us) server_negotiation.cc:407] Connection header received
0927 21:27:54.802812 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:54.802816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:54.802876 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:54.802960 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:54.803700 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:54.804440 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:54.805741 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:54.806186 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:54.806293 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:54.810458 (+ 4165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55187: BlockingRecv error: recv got EOF from 127.0.97.3:55187 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 27 21:27:54 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:54 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:54.899434 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:54.891148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.1:36129)
0927 21:27:54.891709 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:54.891734 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:54.891849 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:54.892160 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:54.892163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:54.892542 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:54.892685 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:54.892693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:54.893795 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:54.893799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:54.894670 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:54.894681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:54.895440 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:54.895446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:54.895617 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:54.896256 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:54.896274 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:54.899185 (+ 2911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":327,"threads_started":1}
W20260927 21:27:54.899821 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:54.891282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36129 (local address 127.0.97.62:41753)
0927 21:27:54.891454 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:54.891458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:54.891822 (+ 364us) server_negotiation.cc:407] Connection header received
0927 21:27:54.891955 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:54.891958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:54.892023 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:54.892122 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:54.892846 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:54.893683 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:54.894828 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:54.895253 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:54.895423 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:54.899387 (+ 3964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36129: BlockingRecv error: recv got EOF from 127.0.97.1:36129 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 21:27:55 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:55 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:55.346705 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:55.338039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.2:59473)
0927 21:27:55.338571 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:55.338586 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:55.338744 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:55.339076 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:55.339080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:55.339478 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:55.339624 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:55.339632 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:55.340688 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:55.340696 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:55.341652 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:55.341664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:55.342362 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:55.342368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:55.342543 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:55.343162 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:55.343179 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:55.346469 (+ 3290us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":329,"threads_started":1}
W20260927 21:27:55.346944 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:55.338143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59473 (local address 127.0.97.62:41753)
0927 21:27:55.338350 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:55.338358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:55.338681 (+ 323us) server_negotiation.cc:407] Connection header received
0927 21:27:55.338874 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:55.338886 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:55.338946 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:55.339055 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:55.339796 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:55.340554 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:55.341799 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:55.342199 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:55.342345 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:55.346652 (+ 4307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59473: BlockingRecv error: recv got EOF from 127.0.97.2:59473 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 21:27:55 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:55 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:55.824038 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:55.811796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.3:57181)
0927 21:27:55.812355 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:55.812372 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:55.812512 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:55.812888 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:55.812892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:55.813316 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:55.813477 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:55.813487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:55.814601 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:55.814606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:55.815548 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:55.815561 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:55.816629 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:55.816635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:55.816832 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:55.820955 (+ 4123us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:55.820976 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:55.823791 (+ 2815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":318,"threads_started":1}
W20260927 21:27:55.824468 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:55.811879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57181 (local address 127.0.97.62:41753)
0927 21:27:55.812085 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:55.812090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:55.812479 (+ 389us) server_negotiation.cc:407] Connection header received
0927 21:27:55.812623 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:55.812627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:55.812698 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:55.812827 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:55.813668 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:55.814443 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:55.815726 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:55.816430 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:55.816545 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:55.824165 (+ 7620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57181: BlockingRecv error: recv got EOF from 127.0.97.3:57181 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 27 21:27:55 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:55 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:55.909332 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:55.900346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.1:54075)
0927 21:27:55.901009 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:55.901026 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:55.901139 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:55.901500 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:55.901504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:55.902026 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:55.902270 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:55.902283 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:55.903715 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:55.903721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:55.904802 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:55.904815 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:55.905477 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:55.905481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:55.905591 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:55.906287 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:55.906306 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:55.909087 (+ 2781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":286,"threads_started":1}
W20260927 21:27:55.909534 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:55.900394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54075 (local address 127.0.97.62:41753)
0927 21:27:55.900655 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:55.900660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:55.901133 (+ 473us) server_negotiation.cc:407] Connection header received
0927 21:27:55.901250 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:55.901255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:55.901325 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:55.901428 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:55.902479 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:55.903531 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:55.904949 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:55.905351 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:55.905445 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:55.909314 (+ 3869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54075: BlockingRecv error: recv got EOF from 127.0.97.1:54075 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Sep 27 21:27:56 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:56 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:56.356635 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:56.347892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.2:49829)
0927 21:27:56.348505 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:56.348521 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:56.348692 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:56.349088 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:56.349093 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:56.349606 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:56.349863 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:56.349873 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:56.351024 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:56.351031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:56.351949 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:56.351961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:56.352673 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:56.352679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:56.352861 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:56.353441 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:56.353458 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:56.356291 (+ 2833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":277,"threads_started":1}
W20260927 21:27:56.356741 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:56.347980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49829 (local address 127.0.97.62:41753)
0927 21:27:56.348189 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:56.348215 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:56.348622 (+ 407us) server_negotiation.cc:407] Connection header received
0927 21:27:56.348885 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:56.348889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:56.348951 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:56.349061 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:56.350013 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:56.350889 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:56.352095 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:56.352518 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:56.352656 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:56.356493 (+ 3837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49829: BlockingRecv error: recv got EOF from 127.0.97.2:49829 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 21:27:56 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:56 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:56.832966 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:56.824973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.3:56769)
0927 21:27:56.825570 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:56.825585 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:56.825784 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:56.826116 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:56.826120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:56.826498 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:56.826660 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:56.826668 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:56.827624 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:56.827628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:56.828544 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:56.828556 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:56.829237 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:56.829241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:56.829364 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:56.829940 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:56.829957 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:56.832711 (+ 2754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":268,"threads_started":1}
W20260927 21:27:56.833225 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:56.825048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56769 (local address 127.0.97.62:41753)
0927 21:27:56.825240 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:56.825247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:56.825745 (+ 498us) server_negotiation.cc:407] Connection header received
0927 21:27:56.825927 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:56.825930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:56.825993 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:56.826069 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:56.826782 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:56.827492 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:56.828664 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:56.829106 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:56.829200 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:56.832953 (+ 3753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56769: BlockingRecv error: recv got EOF from 127.0.97.3:56769 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 21:27:56 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:56 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:56.918977 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:56.910319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.1:60203)
0927 21:27:56.910857 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:56.910878 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:56.911033 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:56.911405 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:56.911411 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:56.911951 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:56.912106 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:56.912116 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:56.913169 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:56.913173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:56.914191 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:56.914203 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:56.914985 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:56.914990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:56.915108 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:56.915733 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:56.915751 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:56.918750 (+ 2999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":445,"threads_started":1}
W20260927 21:27:56.919229 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:56.910551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60203 (local address 127.0.97.62:41753)
0927 21:27:56.910758 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:56.910763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:56.910970 (+ 207us) server_negotiation.cc:407] Connection header received
0927 21:27:56.911158 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:56.911164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:56.911250 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:56.911329 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:56.912246 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:56.913023 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:56.914325 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:56.914793 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:56.914909 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:56.918953 (+ 4044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60203: BlockingRecv error: recv got EOF from 127.0.97.1:60203 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 21:27:57 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:57 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:57.367164 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:57.357831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.2:56569)
0927 21:27:57.358383 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:57.358400 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:57.358526 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:57.358864 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:57.358868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:57.359303 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:57.359458 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:57.359467 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:57.360733 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:57.360738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:57.362008 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:57.362022 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:57.363021 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:57.363027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:57.363204 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:57.363894 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:57.363913 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:57.366910 (+ 2997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":277,"threads_started":1}
W20260927 21:27:57.367408 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:57.358119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56569 (local address 127.0.97.62:41753)
0927 21:27:57.358352 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:57.358385 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:57.358484 (+ 99us) server_negotiation.cc:407] Connection header received
0927 21:27:57.358629 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:57.358633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:57.358696 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:57.358782 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:57.359595 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:57.360567 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:57.362171 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:57.362827 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:57.362963 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:57.367136 (+ 4173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56569: BlockingRecv error: recv got EOF from 127.0.97.2:56569 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 27 21:27:57 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:57 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:57.843415 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:57.834093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.3:48873)
0927 21:27:57.834697 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:57.834713 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:57.834928 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:57.835288 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:57.835292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:57.835709 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:57.835868 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:57.835877 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:57.837260 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:57.837266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:57.838533 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:57.838550 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:57.839391 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:57.839395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:57.839512 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:57.840177 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:57.840195 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:57.843089 (+ 2894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":338,"threads_started":1}
W20260927 21:27:57.843585 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:57.834270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48873 (local address 127.0.97.62:41753)
0927 21:27:57.834559 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:57.834566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:57.834853 (+ 287us) server_negotiation.cc:407] Connection header received
0927 21:27:57.835060 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:57.835066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:57.835145 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:57.835269 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:57.836026 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:57.836988 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:57.838710 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:57.839205 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:57.839312 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:57.843311 (+ 3999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:48873: BlockingRecv error: recv got EOF from 127.0.97.3:48873 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 27 21:27:57 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:57 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:57.929453 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:57.920165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.1:36099)
0927 21:27:57.920748 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:57.920765 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:57.920953 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:57.921325 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:57.921330 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:57.921812 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:57.921988 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:57.921997 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:57.923022 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:57.923029 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:57.924151 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:57.924163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:57.925009 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:57.925014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:57.925193 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:57.925956 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:57.925974 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:57.929188 (+ 3214us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":301,"threads_started":1}
W20260927 21:27:57.929816 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:57.920320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36099 (local address 127.0.97.62:41753)
0927 21:27:57.920615 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:57.920623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:57.920863 (+ 240us) server_negotiation.cc:407] Connection header received
0927 21:27:57.921083 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:57.921088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:57.921168 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:57.921278 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:57.922146 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:57.922877 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:57.924292 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:57.924845 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:57.924979 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:57.929456 (+ 4477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36099: BlockingRecv error: recv got EOF from 127.0.97.1:36099 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Sep 27 21:27:58 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:58 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:58.377260 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:58.368139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.2:38569)
0927 21:27:58.368637 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:58.368653 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:58.368821 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:58.369117 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:58.369121 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:58.369494 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:58.369758 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:58.369770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:58.370759 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:58.370763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:58.371729 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:58.371739 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:58.372554 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:58.372564 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:58.372719 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:58.373365 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:58.373383 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:58.377000 (+ 3617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":256,"threads_started":1}
W20260927 21:27:58.377485 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:58.368281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38569 (local address 127.0.97.62:41753)
0927 21:27:58.368507 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:58.368514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:58.368740 (+ 226us) server_negotiation.cc:407] Connection header received
0927 21:27:58.368929 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:58.368933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:58.368996 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:58.369080 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:58.369893 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:58.370617 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:58.371889 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:58.372374 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:58.372492 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:58.377199 (+ 4707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38569: BlockingRecv error: recv got EOF from 127.0.97.2:38569 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 27 21:27:58 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:58 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:58.853129 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:58.844492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.3:54773)
0927 21:27:58.845021 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:58.845037 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:58.845159 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:58.845467 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:58.845470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:58.845889 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:58.846062 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:58.846070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:58.847076 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:58.847080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:58.848065 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:58.848077 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:58.848860 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:58.848864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:58.848993 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:58.849675 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:58.849693 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:58.852889 (+ 3196us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":302,"threads_started":1}
W20260927 21:27:58.853372 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:58.844678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54773 (local address 127.0.97.62:41753)
0927 21:27:58.844933 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:58.844939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:58.845125 (+ 186us) server_negotiation.cc:407] Connection header received
0927 21:27:58.845272 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:58.845276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:58.845338 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:58.845423 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:58.846192 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:58.846884 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:58.848247 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:58.848683 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:58.848780 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:58.853079 (+ 4299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:54773: BlockingRecv error: recv got EOF from 127.0.97.3:54773 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Sep 27 21:27:58 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:58 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:58.939211 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:58.930573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.1:55823)
0927 21:27:58.931142 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:58.931158 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:58.931319 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:58.931660 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:58.931665 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:58.932067 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:58.932209 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:58.932218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:58.933241 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:58.933249 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:58.934261 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:58.934274 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:58.934998 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:58.935003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:58.935182 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:58.935824 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:58.935851 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:58.938881 (+ 3030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":324,"threads_started":1}
W20260927 21:27:58.939498 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:58.930653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55823 (local address 127.0.97.62:41753)
0927 21:27:58.930934 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:58.930943 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:58.931252 (+ 309us) server_negotiation.cc:407] Connection header received
0927 21:27:58.931448 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:58.931453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:58.931522 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:58.931631 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:58.932363 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:58.933103 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:58.934405 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:58.934842 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:58.934989 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:58.939077 (+ 4088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55823: BlockingRecv error: recv got EOF from 127.0.97.1:55823 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 27 21:27:59 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:59 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:59.386587 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:59.378202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.2:41659)
0927 21:27:59.378699 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:59.378725 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:59.378841 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:59.379205 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:59.379210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:59.379563 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:59.379713 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:59.379721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:59.380794 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:59.380798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:59.381881 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:59.381894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:59.382645 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:59.382650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:59.382805 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:59.383441 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:59.383464 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:59.386357 (+ 2893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":300,"threads_started":1}
W20260927 21:27:59.386878 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:59.378322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41659 (local address 127.0.97.62:41753)
0927 21:27:59.378589 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:59.378596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:59.378812 (+ 216us) server_negotiation.cc:407] Connection header received
0927 21:27:59.378927 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:59.378930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:59.378991 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:59.379078 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:59.379886 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:59.380617 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:59.382051 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:59.382486 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:59.382592 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:59.386578 (+ 3986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41659: BlockingRecv error: recv got EOF from 127.0.97.2:41659 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 27 21:27:59 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:59 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:59.862840 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:59.854307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.3:49033)
0927 21:27:59.854861 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:59.854878 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:59.854991 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:59.855366 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:59.855369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:59.855719 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:59.855857 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:59.855865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:59.856947 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:59.856955 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:59.858030 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:59.858043 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:59.858712 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:59.858716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:59.858832 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:59.859460 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:59.859478 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:59.862608 (+ 3130us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":317,"threads_started":1}
W20260927 21:27:59.863122 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:59.854475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49033 (local address 127.0.97.62:41753)
0927 21:27:59.854652 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:59.854656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:59.854954 (+ 298us) server_negotiation.cc:407] Connection header received
0927 21:27:59.855136 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:59.855141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:59.855223 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:59.855326 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:59.856019 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:59.856810 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:59.858168 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:59.858584 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:59.858681 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:59.862838 (+ 4157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49033: BlockingRecv error: recv got EOF from 127.0.97.3:49033 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 21:27:59 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:27:59 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:27:59.949182 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:27:59.940290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.1:33163)
0927 21:27:59.940840 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 21:27:59.940856 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:27:59.940974 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:27:59.941391 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:27:59.941395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:27:59.941892 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:27:59.942062 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:59.942070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:59.943112 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:59.943116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:59.944036 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:27:59.944048 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:27:59.944777 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:27:59.944782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:27:59.944894 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:59.945498 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:27:59.945526 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:27:59.948826 (+ 3300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":308,"threads_started":1}
W20260927 21:27:59.949321 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:27:59.940371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33163 (local address 127.0.97.62:41753)
0927 21:27:59.940591 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 21:27:59.940598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:27:59.940939 (+ 341us) server_negotiation.cc:407] Connection header received
0927 21:27:59.941159 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:27:59.941164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:27:59.941235 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:27:59.941326 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:27:59.942211 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:59.942967 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:59.944185 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:27:59.944605 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:27:59.944704 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:27:59.949008 (+ 4304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33163: BlockingRecv error: recv got EOF from 127.0.97.1:33163 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 21:28:00 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:28:00 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:28:00.396548 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:00.387705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.2:41049)
0927 21:28:00.388480 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0927 21:28:00.388497 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:28:00.388662 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:28:00.389018 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:28:00.389024 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:28:00.389448 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:28:00.389655 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:00.389664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:00.390673 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:00.390682 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:00.391582 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:00.391593 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:00.392301 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:00.392307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:00.392495 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:00.393168 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:28:00.393186 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:28:00.396202 (+ 3016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":546,"threads_started":1}
W20260927 21:28:00.396915 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:28:00.387983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41049 (local address 127.0.97.62:41753)
0927 21:28:00.388214 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:00.388224 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:00.388593 (+ 369us) server_negotiation.cc:407] Connection header received
0927 21:28:00.388802 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:00.388809 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:00.388877 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:00.388984 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:00.389800 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:00.390531 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:00.391729 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:00.392142 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:00.392270 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:00.396612 (+ 4342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41049: BlockingRecv error: recv got EOF from 127.0.97.2:41049 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 21:28:00 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:28:00 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:28:00.878198 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:00.863735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.3:42481)
0927 21:28:00.864169 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 21:28:00.864184 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:28:00.864295 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:28:00.864558 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:28:00.864561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:28:00.864984 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:28:00.865145 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:00.865154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:00.866291 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:00.866295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:00.867251 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:00.867263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:00.868007 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:00.868012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:00.868132 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:00.874816 (+ 6684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:28:00.874839 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:28:00.877959 (+ 3120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":236,"threads_started":1}
W20260927 21:28:00.878430 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:28:00.863795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42481 (local address 127.0.97.62:41753)
0927 21:28:00.864012 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:00.864018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:00.864260 (+ 242us) server_negotiation.cc:407] Connection header received
0927 21:28:00.864376 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:00.864380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:00.864440 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:00.864523 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:00.865258 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:00.866145 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:00.867400 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:00.867834 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:00.867952 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:00.878179 (+ 10227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42481: BlockingRecv error: recv got EOF from 127.0.97.3:42481 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 21:28:00 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:28:00 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:28:00.958846 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:00.950223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.1:55115)
0927 21:28:00.950789 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0927 21:28:00.950806 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:28:00.950961 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:28:00.951326 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:28:00.951330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:28:00.951738 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:28:00.951898 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:00.951907 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:00.952911 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:00.952917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:00.954021 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:00.954041 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:00.954793 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:00.954799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:00.955004 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:00.955589 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:28:00.955606 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:28:00.958489 (+ 2883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":303,"threads_started":1}
W20260927 21:28:00.958987 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:28:00.950323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55115 (local address 127.0.97.62:41753)
0927 21:28:00.950517 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:00.950526 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:00.950892 (+ 366us) server_negotiation.cc:407] Connection header received
0927 21:28:00.951094 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:00.951098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:00.951186 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:00.951297 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:00.952037 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:00.952771 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:00.954172 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:00.954642 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:00.954777 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:00.958691 (+ 3914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55115: BlockingRecv error: recv got EOF from 127.0.97.1:55115 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 21:28:01 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:28:01 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:28:01.406700 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:01.397742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.2:45493)
0927 21:28:01.398324 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0927 21:28:01.398340 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:28:01.398462 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:28:01.398820 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:28:01.398824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:28:01.399200 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:28:01.399424 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:01.399437 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:01.400785 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:01.400791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:01.401992 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:01.402006 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:01.402693 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:01.402714 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:01.402848 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:01.403439 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:28:01.403456 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:28:01.406466 (+ 3010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":330,"threads_started":1}
W20260927 21:28:01.406956 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:28:01.397758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45493 (local address 127.0.97.62:41753)
0927 21:28:01.397949 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:01.397958 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:01.398425 (+ 467us) server_negotiation.cc:407] Connection header received
0927 21:28:01.398583 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:01.398589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:01.398682 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:01.398793 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:01.399624 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:01.400628 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:01.402128 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:01.402548 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:01.402647 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:01.406697 (+ 4050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:45493: BlockingRecv error: recv got EOF from 127.0.97.2:45493 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 21:28:01 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:28:01 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:28:01.888317 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:01.879356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.3:36541)
0927 21:28:01.879943 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0927 21:28:01.879958 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:28:01.880137 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:28:01.880475 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:28:01.880480 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:28:01.880893 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:28:01.881042 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:01.881051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:01.882130 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:01.882138 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:01.883182 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:01.883194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:01.883972 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:01.883977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:01.884156 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:01.884954 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:28:01.884973 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:28:01.887943 (+ 2970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":341,"threads_started":1}
W20260927 21:28:01.888492 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:28:01.879491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36541 (local address 127.0.97.62:41753)
0927 21:28:01.879750 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:01.879758 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:01.880052 (+ 294us) server_negotiation.cc:407] Connection header received
0927 21:28:01.880266 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:01.880271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:01.880339 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:01.880447 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:01.881190 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:01.881988 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:01.883345 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:01.883820 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:01.883951 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:01.888214 (+ 4263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36541: BlockingRecv error: recv got EOF from 127.0.97.3:36541 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 27 21:28:01 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:28:01 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:28:01.968274 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:01.959911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.1:54333)
0927 21:28:01.960435 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 21:28:01.960451 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:28:01.960559 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:28:01.960860 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:28:01.960863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:28:01.961234 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:28:01.961403 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:01.961412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:01.962460 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:01.962466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:01.963543 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:01.963555 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:01.964278 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:01.964282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:01.964392 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:01.965084 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:28:01.965102 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:28:01.968040 (+ 2938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":270,"threads_started":1}
W20260927 21:28:01.968488 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:28:01.959961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54333 (local address 127.0.97.62:41753)
0927 21:28:01.960186 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:01.960195 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:01.960526 (+ 331us) server_negotiation.cc:407] Connection header received
0927 21:28:01.960651 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:01.960655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:01.960730 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:01.960827 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:01.961516 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:01.962295 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:01.963703 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:01.964132 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:01.964224 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:01.968263 (+ 4039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54333: BlockingRecv error: recv got EOF from 127.0.97.1:54333 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 21:28:02 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:28:02 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:28:02.415984 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:02.407810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.2:40293)
0927 21:28:02.408358 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0927 21:28:02.408379 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:28:02.408518 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:28:02.408834 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:28:02.408837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:28:02.409218 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:28:02.409391 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:02.409400 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:02.410403 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:02.410407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:02.411329 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:02.411340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:02.412073 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:02.412078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:02.412210 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:02.412928 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:28:02.412946 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:28:02.415748 (+ 2802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":445,"threads_started":1}
W20260927 21:28:02.416206 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:28:02.407901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40293 (local address 127.0.97.62:41753)
0927 21:28:02.408091 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:02.408098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:02.408453 (+ 355us) server_negotiation.cc:407] Connection header received
0927 21:28:02.408633 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:02.408636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:02.408710 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:02.408790 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:02.409518 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:02.410260 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:02.411479 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:02.411907 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:02.412003 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:02.415966 (+ 3963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40293: BlockingRecv error: recv got EOF from 127.0.97.2:40293 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 21:28:02 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:28:02 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:28:02.898062 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:02.889431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.3:60081)
0927 21:28:02.890011 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0927 21:28:02.890028 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:28:02.890154 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:28:02.890533 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:28:02.890537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:28:02.890980 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:28:02.891131 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:02.891140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:02.892200 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:02.892207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:02.893137 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:02.893148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:02.893970 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:02.893977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:02.894139 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:02.894783 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:28:02.894801 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:28:02.897714 (+ 2913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":331,"threads_started":1}
W20260927 21:28:02.898162 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:28:02.889531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60081 (local address 127.0.97.62:41753)
0927 21:28:02.889751 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:02.889756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:02.890148 (+ 392us) server_negotiation.cc:407] Connection header received
0927 21:28:02.890275 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:02.890280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:02.890376 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:02.890465 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:02.891304 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:02.892058 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:02.893293 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:02.893777 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:02.893904 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:02.897926 (+ 4022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60081: BlockingRecv error: recv got EOF from 127.0.97.3:60081 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 21:28:02 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:28:02 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:28:02.977506 9701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:02.969216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.1:56743)
0927 21:28:02.969782 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0927 21:28:02.969800 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:28:02.969967 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:28:02.970325 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:28:02.970330 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:28:02.970728 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:28:02.970898 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:02.970908 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:02.971942 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:02.971949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:02.972884 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:02.972895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:02.973591 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:02.973596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:02.973796 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:02.974414 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:28:02.974432 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:28:02.977254 (+ 2822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":287,"threads_started":1}
W20260927 21:28:02.977869 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:28:02.969276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56743 (local address 127.0.97.62:41753)
0927 21:28:02.969654 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:02.969660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:02.969893 (+ 233us) server_negotiation.cc:407] Connection header received
0927 21:28:02.970093 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:02.970099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:02.970186 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:02.970296 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:02.971045 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:02.971798 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:02.973044 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:02.973440 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:02.973577 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:02.977489 (+ 3912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56743: BlockingRecv error: recv got EOF from 127.0.97.1:56743 (error 108)
Metrics: {"server-negotiator.queue_time_us":210}
Sep 27 21:28:03 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:28:03 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:28:03.425529 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:03.417017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.2:58683)
0927 21:28:03.417501 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 21:28:03.417518 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:28:03.417699 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:28:03.418062 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:28:03.418067 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:28:03.418494 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:28:03.418677 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:03.418687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:03.419725 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:03.419731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:03.420671 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:03.420683 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:03.421409 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:03.421415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:03.421676 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:03.422351 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:28:03.422368 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:28:03.425277 (+ 2909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":255,"threads_started":1}
W20260927 21:28:03.425731 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:28:03.417089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58683 (local address 127.0.97.62:41753)
0927 21:28:03.417316 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:03.417324 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:03.417603 (+ 279us) server_negotiation.cc:407] Connection header received
0927 21:28:03.417840 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:03.417846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:03.417918 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:03.418029 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:03.418845 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:03.419571 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:03.420821 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:03.421249 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:03.421383 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:03.425497 (+ 4114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58683: BlockingRecv error: recv got EOF from 127.0.97.2:58683 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 21:28:03 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:28:03 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:28:03.907878 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:03.899156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.3:38643)
0927 21:28:03.899737 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0927 21:28:03.899760 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:28:03.899916 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:28:03.900335 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:28:03.900340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:28:03.900822 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:28:03.901066 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:03.901079 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:03.902365 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:03.902369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:03.903267 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:03.903288 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:03.903984 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:03.903987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:03.904102 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:03.904788 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:28:03.904807 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:28:03.907627 (+ 2820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":307,"threads_started":1}
W20260927 21:28:03.908195 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:28:03.899260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38643 (local address 127.0.97.62:41753)
0927 21:28:03.899475 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:03.899484 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:03.899854 (+ 370us) server_negotiation.cc:407] Connection header received
0927 21:28:03.900053 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:03.900060 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:03.900145 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:03.900266 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:03.901219 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:03.902218 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:03.903411 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:03.903861 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:03.903956 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:03.907875 (+ 3919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:38643: BlockingRecv error: recv got EOF from 127.0.97.3:38643 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 21:28:03 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:28:03 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:28:03.987525 9704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:03.978645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.1:59491)
0927 21:28:03.979145 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 21:28:03.979162 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:28:03.979288 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:28:03.979633 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:28:03.979636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:28:03.980044 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:28:03.980242 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:03.980252 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:03.981301 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:03.981305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:03.982304 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:03.982317 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:03.983071 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:03.983076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:03.983192 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:03.983936 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:28:03.983953 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:28:03.987157 (+ 3204us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":293,"threads_started":1}
W20260927 21:28:03.987607 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:28:03.978813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59491 (local address 127.0.97.62:41753)
0927 21:28:03.979057 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:03.979064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:03.979301 (+ 237us) server_negotiation.cc:407] Connection header received
0927 21:28:03.979397 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:03.979401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:03.979485 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:03.979564 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:03.980429 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:03.981156 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:03.982451 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:03.982930 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:03.983028 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:03.987364 (+ 4336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59491: BlockingRecv error: recv got EOF from 127.0.97.1:59491 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 27 21:28:04 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:28:04 dist-test-slave-bxm0 krb5kdc[8658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:28:04.435252 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:04.426724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41753 (local address 127.0.97.2:50619)
0927 21:28:04.427237 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 21:28:04.427253 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:28:04.427368 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:28:04.427647 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:28:04.427650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:28:04.428073 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:28:04.428263 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:04.428271 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:04.429300 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:04.429305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:04.430309 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:04.430322 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:04.431145 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:04.431151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:04.431358 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:04.432131 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:28:04.432149 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:28:04.434991 (+ 2842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41753: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":297,"threads_started":1}
W20260927 21:28:04.435451 9395 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:28:04.426765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50619 (local address 127.0.97.62:41753)
0927 21:28:04.426946 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:04.426950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:04.427323 (+ 373us) server_negotiation.cc:407] Connection header received
0927 21:28:04.427464 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:04.427468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:04.427526 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:04.427608 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:04.428397 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:04.429136 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:04.430433 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:04.430966 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:04.431115 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:04.435208 (+ 4093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50619: BlockingRecv error: recv got EOF from 127.0.97.2:50619 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 21:28:04.732995 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 9246
I20260927 21:28:04.745110 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:04.734151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35824 (local address 127.0.97.1:44145)
0927 21:28:04.734718 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:04.734725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:04.734753 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:28:04.734812 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:04.734818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:04.735039 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:04.735213 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:04.736599 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:04.737664 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:04.738896 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:04.739497 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:04.739607 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:04.742182 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:28:04.742191 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:28:04.742212 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:28:04.742246 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:28:04.744627 (+ 2381us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 21:28:04.744886 (+ 259us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35824: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":300,"threads_started":1}
W20260927 21:28:04.745261 9707 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35824: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 21:28:04.745357 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 9246: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:44145: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 21:28:04.745582 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 9246
I20260927 21:28:04.762262 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 9397
I20260927 21:28:04.775589 9708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:04.763552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46160 (local address 127.0.97.2:33731)
0927 21:28:04.764155 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:04.764162 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:04.764192 (+ 30us) server_negotiation.cc:407] Connection header received
0927 21:28:04.764253 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:04.764259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:04.764476 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:04.764626 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:04.766105 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:04.767210 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:04.768464 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:04.769051 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:04.769175 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:04.771726 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:28:04.771735 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:28:04.771749 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:28:04.771788 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:28:04.774989 (+ 3201us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 21:28:04.775275 (+ 286us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46160: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":305,"threads_started":1}
W20260927 21:28:04.775714 9708 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46160: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 21:28:04.775755 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 9397: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:33731: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 21:28:04.775936 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 9397
I20260927 21:28:04.792636 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 9529
I20260927 21:28:04.805601 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:04.793792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41842 (local address 127.0.97.3:39943)
0927 21:28:04.794448 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:04.794455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:04.794483 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:28:04.794547 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:04.794554 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:04.794784 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:04.794944 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:04.796445 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:04.797656 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:04.799188 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:04.799824 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:04.799948 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:04.802572 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:28:04.802581 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:28:04.802595 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:28:04.802637 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:28:04.805066 (+ 2429us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 21:28:04.805339 (+ 273us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41842: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":361,"threads_started":1}
W20260927 21:28:04.805801 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 9529: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:39943: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 21:28:04.805903 9709 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41842: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 21:28:04.805990 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 9529
I20260927 21:28:04.822093 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 9177
I20260927 21:28:04.823475 9239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:28:04.937155 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 9177
2026-09-27T21:28:04Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21159 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T21:28:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:28:04Z Disabled control of system clock
I20260927 21:28:05.013322 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40069
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40069
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 21:28:05.345947 9714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:28:05.346254 9714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:28:05.346333 9714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:28:05.353643 9714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:28:05.353761 9714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:28:05.353799 9714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:28:05.353848 9714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:28:05.353900 9714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:28:05.363039 9714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40069
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:28:05.364305 9714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:28:05.366137 9714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:28:05.378894 9720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:28:05.379108 9722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:28:05.378945 9719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:28:05.379928 9714 server_base.cc:1034] running on GCE node
I20260927 21:28:05.380736 9714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:28:05.382236 9714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:28:05.383463 9714 hybrid_clock.cc:648] HybridClock initialized: now 1790544485383407 us; error 59 us; skew 500 ppm
I20260927 21:28:05.383814 9714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:28:05.386247 9714 webserver.cc:460] Webserver started at http://127.0.97.62:33959/ using document root <none> and password file <none>
I20260927 21:28:05.386981 9714 fs_manager.cc:359] Metadata directory not provided
I20260927 21:28:05.387115 9714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:28:05.387449 9714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:28:05.390545 9714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "e0c0b40701b74966bbb708ea6ff5cb46"
format_stamp: "Formatted at 2026-09-27 21:28:05 on dist-test-slave-bxm0"
server_key: "c0e69dc324f065291b9e9e07accbfc25"
server_key_iv: "c9cddcd3fb802707473214e5c1616da9"
server_key_version: "encryptionkey@0"
I20260927 21:28:05.391374 9714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "e0c0b40701b74966bbb708ea6ff5cb46"
format_stamp: "Formatted at 2026-09-27 21:28:05 on dist-test-slave-bxm0"
server_key: "c0e69dc324f065291b9e9e07accbfc25"
server_key_iv: "c9cddcd3fb802707473214e5c1616da9"
server_key_version: "encryptionkey@0"
I20260927 21:28:05.397007 9714 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:28:05.400796 9728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:05.402303 9714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:28:05.402504 9714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "e0c0b40701b74966bbb708ea6ff5cb46"
format_stamp: "Formatted at 2026-09-27 21:28:05 on dist-test-slave-bxm0"
server_key: "c0e69dc324f065291b9e9e07accbfc25"
server_key_iv: "c9cddcd3fb802707473214e5c1616da9"
server_key_version: "encryptionkey@0"
I20260927 21:28:05.402737 9714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:28:05.420328 9714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:28:05.421258 9714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:28:05.421569 9714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:28:05.440421 9714 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40069
I20260927 21:28:05.440439 9779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40069 every 8 connection(s)
I20260927 21:28:05.441931 9714 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:28:05.445897 9780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:28:05.446259 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 9714
I20260927 21:28:05.446444 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:28:05.447247 388 external_mini_cluster.cc:1428] Setting key eaccb7e90eda4f0331b4b42d86e1d60f
I20260927 21:28:05.460868 9780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0c0b40701b74966bbb708ea6ff5cb46: Bootstrap starting.
I20260927 21:28:05.461797 9783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:05.449456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51370 (local address 127.0.97.62:40069)
0927 21:28:05.450574 (+ 1118us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:05.450585 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:05.450632 (+ 47us) server_negotiation.cc:407] Connection header received
0927 21:28:05.452221 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:05.452244 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:05.452794 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:05.453166 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:05.453855 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:05.455137 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:05.457014 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:05.457924 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:05.458084 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:05.459316 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:28:05.459323 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:28:05.459339 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:28:05.459889 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:28:05.459972 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:28:05.459997 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:28:05.460085 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:05.460332 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 21:28:05.460698 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":457,"threads_started":1}
I20260927 21:28:05.464586 9780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0c0b40701b74966bbb708ea6ff5cb46: Neither blocks nor log segments found. Creating new log.
I20260927 21:28:05.465698 9780 log.cc:824] T 00000000000000000000000000000000 P e0c0b40701b74966bbb708ea6ff5cb46: Log is configured to *not* fsync() on all Append() calls
I20260927 21:28:05.468648 9780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0c0b40701b74966bbb708ea6ff5cb46: No bootstrap required, opened a new log
I20260927 21:28:05.476331 9780 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0c0b40701b74966bbb708ea6ff5cb46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c0b40701b74966bbb708ea6ff5cb46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40069 } }
I20260927 21:28:05.476718 9780 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0c0b40701b74966bbb708ea6ff5cb46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:28:05.476792 9780 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0c0b40701b74966bbb708ea6ff5cb46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0c0b40701b74966bbb708ea6ff5cb46, State: Initialized, Role: FOLLOWER
I20260927 21:28:05.477385 9780 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0c0b40701b74966bbb708ea6ff5cb46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c0b40701b74966bbb708ea6ff5cb46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40069 } }
I20260927 21:28:05.477588 9780 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0c0b40701b74966bbb708ea6ff5cb46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:28:05.477754 9780 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0c0b40701b74966bbb708ea6ff5cb46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:28:05.477893 9780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0c0b40701b74966bbb708ea6ff5cb46 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:28:05.480046 9780 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0c0b40701b74966bbb708ea6ff5cb46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c0b40701b74966bbb708ea6ff5cb46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40069 } }
I20260927 21:28:05.480583 9780 leader_election.cc:302] T 00000000000000000000000000000000 P e0c0b40701b74966bbb708ea6ff5cb46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0c0b40701b74966bbb708ea6ff5cb46; no voters:
I20260927 21:28:05.481137 9780 leader_election.cc:288] T 00000000000000000000000000000000 P e0c0b40701b74966bbb708ea6ff5cb46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:28:05.481328 9785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0c0b40701b74966bbb708ea6ff5cb46 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:28:05.482131 9785 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0c0b40701b74966bbb708ea6ff5cb46 [term 1 LEADER]: Becoming Leader. State: Replica: e0c0b40701b74966bbb708ea6ff5cb46, State: Running, Role: LEADER
I20260927 21:28:05.482718 9785 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0c0b40701b74966bbb708ea6ff5cb46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c0b40701b74966bbb708ea6ff5cb46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40069 } }
I20260927 21:28:05.483379 9780 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0c0b40701b74966bbb708ea6ff5cb46 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:28:05.485925 9787 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c0b40701b74966bbb708ea6ff5cb46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0c0b40701b74966bbb708ea6ff5cb46. Latest consensus state: current_term: 1 leader_uuid: "e0c0b40701b74966bbb708ea6ff5cb46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c0b40701b74966bbb708ea6ff5cb46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40069 } } }
I20260927 21:28:05.486371 9787 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c0b40701b74966bbb708ea6ff5cb46 [sys.catalog]: This master's current role is: LEADER
I20260927 21:28:05.486218 9786 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c0b40701b74966bbb708ea6ff5cb46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0c0b40701b74966bbb708ea6ff5cb46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c0b40701b74966bbb708ea6ff5cb46" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40069 } } }
I20260927 21:28:05.486521 9786 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c0b40701b74966bbb708ea6ff5cb46 [sys.catalog]: This master's current role is: LEADER
I20260927 21:28:05.488015 9794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:28:05.493755 9794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:28:05.502974 9794 catalog_manager.cc:1347] Generated new cluster ID: aa58aeca163d4c109e26060be232313d
I20260927 21:28:05.503106 9794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:28:05.531222 9794 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:28:05.532681 9794 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:28:05.541208 9794 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0c0b40701b74966bbb708ea6ff5cb46: Generated new TSK 0
I20260927 21:28:05.542152 9794 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:28:05.557696 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40069
--builtin_ntp_servers=127.0.97.20:39329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 21:28:05.906884 9804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:28:05.907183 9804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:28:05.907315 9804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:28:05.915136 9804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:28:05.915256 9804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:28:05.915361 9804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:28:05.924993 9804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:28:05.926316 9804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:28:05.928020 9804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:28:05.941526 9809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:28:05.941561 9810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:28:05.941494 9812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:28:05.942408 9804 server_base.cc:1034] running on GCE node
I20260927 21:28:05.943003 9804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:28:05.943902 9804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:28:05.945143 9804 hybrid_clock.cc:648] HybridClock initialized: now 1790544485945092 us; error 45 us; skew 500 ppm
I20260927 21:28:05.945474 9804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:28:05.951366 9804 webserver.cc:460] Webserver started at http://127.0.97.1:40085/ using document root <none> and password file <none>
I20260927 21:28:05.952186 9804 fs_manager.cc:359] Metadata directory not provided
I20260927 21:28:05.952343 9804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:28:05.952719 9804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:28:05.955699 9804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/data/instance:
uuid: "77b75f313d6c4da19c7b121241179b58"
format_stamp: "Formatted at 2026-09-27 21:28:05 on dist-test-slave-bxm0"
server_key: "f2046ec77396e80f95260e1de960c246"
server_key_iv: "0e2fde179e2f777de88f1b6b84769307"
server_key_version: "encryptionkey@0"
I20260927 21:28:05.956481 9804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/wal/instance:
uuid: "77b75f313d6c4da19c7b121241179b58"
format_stamp: "Formatted at 2026-09-27 21:28:05 on dist-test-slave-bxm0"
server_key: "f2046ec77396e80f95260e1de960c246"
server_key_iv: "0e2fde179e2f777de88f1b6b84769307"
server_key_version: "encryptionkey@0"
I20260927 21:28:05.962991 9804 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260927 21:28:05.966610 9818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:05.968220 9804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:28:05.968420 9804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "77b75f313d6c4da19c7b121241179b58"
format_stamp: "Formatted at 2026-09-27 21:28:05 on dist-test-slave-bxm0"
server_key: "f2046ec77396e80f95260e1de960c246"
server_key_iv: "0e2fde179e2f777de88f1b6b84769307"
server_key_version: "encryptionkey@0"
I20260927 21:28:05.968667 9804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:28:05.981690 9804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:28:05.982566 9804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:28:05.982852 9804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:28:05.983935 9804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:28:05.985888 9804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:28:05.985977 9804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:05.986063 9804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:28:05.986117 9804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:06.021061 9804 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40357
I20260927 21:28:06.021160 9930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40357 every 8 connection(s)
I20260927 21:28:06.022473 9804 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:28:06.024084 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 9804
I20260927 21:28:06.024269 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/wal/instance
I20260927 21:28:06.024914 388 external_mini_cluster.cc:1428] Setting key d82e44ed59bcc225bf0c2437c34ae86c
I20260927 21:28:06.027948 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40069
--builtin_ntp_servers=127.0.97.20:39329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 21:28:06.039176 9933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:06.026929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34711 (local address 127.0.97.62:40069)
0927 21:28:06.027424 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:06.027431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:06.028946 (+ 1515us) server_negotiation.cc:407] Connection header received
0927 21:28:06.030745 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:06.030754 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:06.030856 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:06.030984 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:06.032133 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:06.033341 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:06.035915 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:06.036346 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:06.036460 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:06.038334 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:28:06.038340 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:28:06.038343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:28:06.038363 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:28:06.038404 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:28:06.038417 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:28:06.038499 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:06.038820 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 21:28:06.038913 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":255,"threads_started":1}
I20260927 21:28:06.040704 9934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:06.027652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40069 (local address 127.0.97.1:34711)
0927 21:28:06.028747 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0927 21:28:06.028801 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 21:28:06.030473 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:28:06.031233 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:28:06.031242 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:28:06.031345 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:28:06.031941 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:06.031958 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:06.033499 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:06.033503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:06.035678 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:06.035699 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:06.036603 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:06.036613 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:06.036977 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:06.037923 (+ 946us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:28:06.037929 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:28:06.037993 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:28:06.038106 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:28:06.038525 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:28:06.038538 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:28:06.038549 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 21:28:06.038746 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 21:28:06.039231 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":324,"threads_started":1}
I20260927 21:28:06.042615 9931 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40069
I20260927 21:28:06.042941 9931 heartbeater.cc:464] Registering TS with master...
I20260927 21:28:06.043843 9931 heartbeater.cc:511] Master 127.0.97.62:40069 requested a full tablet report, sending...
I20260927 21:28:06.046315 9745 ts_manager.cc:194] Registered new tserver with Master: 77b75f313d6c4da19c7b121241179b58 (127.0.97.1:40357)
I20260927 21:28:06.048674 9745 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:34711
W20260927 21:28:06.362703 9935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:28:06.362995 9935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:28:06.363080 9935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:28:06.370285 9935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:28:06.370407 9935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:28:06.370513 9935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:28:06.379865 9935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:28:06.381107 9935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:28:06.382787 9935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:28:06.396442 9941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:28:06.396466 9943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:28:06.396495 9940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:28:06.397464 9935 server_base.cc:1034] running on GCE node
I20260927 21:28:06.398226 9935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:28:06.399260 9935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:28:06.400460 9935 hybrid_clock.cc:648] HybridClock initialized: now 1790544486400415 us; error 43 us; skew 500 ppm
I20260927 21:28:06.400810 9935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:28:06.403194 9935 webserver.cc:460] Webserver started at http://127.0.97.2:40875/ using document root <none> and password file <none>
I20260927 21:28:06.403916 9935 fs_manager.cc:359] Metadata directory not provided
I20260927 21:28:06.404049 9935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:28:06.404372 9935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:28:06.407363 9935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/data/instance:
uuid: "0d40a79aace14389bab647618bc72bb0"
format_stamp: "Formatted at 2026-09-27 21:28:06 on dist-test-slave-bxm0"
server_key: "19af93951edbc891ea79a8bae4ba2930"
server_key_iv: "a0c17f4263cda70e6538bd080e0a161e"
server_key_version: "encryptionkey@0"
I20260927 21:28:06.408131 9935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0d40a79aace14389bab647618bc72bb0"
format_stamp: "Formatted at 2026-09-27 21:28:06 on dist-test-slave-bxm0"
server_key: "19af93951edbc891ea79a8bae4ba2930"
server_key_iv: "a0c17f4263cda70e6538bd080e0a161e"
server_key_version: "encryptionkey@0"
I20260927 21:28:06.413183 9935 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 21:28:06.416889 9949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:06.418404 9935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 21:28:06.418598 9935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "0d40a79aace14389bab647618bc72bb0"
format_stamp: "Formatted at 2026-09-27 21:28:06 on dist-test-slave-bxm0"
server_key: "19af93951edbc891ea79a8bae4ba2930"
server_key_iv: "a0c17f4263cda70e6538bd080e0a161e"
server_key_version: "encryptionkey@0"
I20260927 21:28:06.418864 9935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:28:06.440789 9935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:28:06.441728 9935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:28:06.442039 9935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:28:06.443008 9935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:28:06.444622 9935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:28:06.444744 9935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:06.444891 9935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:28:06.444980 9935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:06.479140 9935 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39329
I20260927 21:28:06.479238 10061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39329 every 8 connection(s)
I20260927 21:28:06.480489 9935 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:28:06.484938 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 9935
I20260927 21:28:06.485109 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/wal/instance
I20260927 21:28:06.485533 388 external_mini_cluster.cc:1428] Setting key 3385b9bf34f1e2bbc0538290ce90031a
I20260927 21:28:06.488555 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40069
--builtin_ntp_servers=127.0.97.20:39329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 21:28:06.493842 9933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:06.483073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46787 (local address 127.0.97.62:40069)
0927 21:28:06.483249 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:06.483254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:06.484890 (+ 1636us) server_negotiation.cc:407] Connection header received
0927 21:28:06.487012 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:06.487017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:06.487137 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:06.487260 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:06.488445 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:06.489243 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:06.490959 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:06.491366 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:06.491500 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:06.493003 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:28:06.493008 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:28:06.493011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:28:06.493030 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:28:06.493061 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:28:06.493073 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:28:06.493169 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:06.493405 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 21:28:06.493505 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:28:06.495115 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:06.483601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40069 (local address 127.0.97.2:46787)
0927 21:28:06.484656 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0927 21:28:06.484750 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0927 21:28:06.486735 (+ 1985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:28:06.487510 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:28:06.487519 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:28:06.487618 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:28:06.488262 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:06.488292 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:06.489408 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:06.489413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:06.490799 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:06.490810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:06.491615 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:06.491625 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:06.491860 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:06.492633 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:28:06.492636 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:28:06.492682 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:28:06.492790 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:28:06.493175 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:28:06.493189 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:28:06.493199 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 21:28:06.493397 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 21:28:06.493912 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":269,"threads_started":1}
I20260927 21:28:06.496760 10062 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40069
I20260927 21:28:06.497102 10062 heartbeater.cc:464] Registering TS with master...
I20260927 21:28:06.498099 10062 heartbeater.cc:511] Master 127.0.97.62:40069 requested a full tablet report, sending...
I20260927 21:28:06.499831 9745 ts_manager.cc:194] Registered new tserver with Master: 0d40a79aace14389bab647618bc72bb0 (127.0.97.2:39329)
I20260927 21:28:06.500684 9745 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:46787
W20260927 21:28:06.821674 10065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:28:06.822026 10065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:28:06.822187 10065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:28:06.829499 10065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:28:06.829694 10065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:28:06.829872 10065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:28:06.839226 10065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:28:06.840610 10065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:28:06.842525 10065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:28:06.855789 10070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:28:06.855695 10073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:28:06.855829 10071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:28:06.857369 10065 server_base.cc:1034] running on GCE node
I20260927 21:28:06.858028 10065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:28:06.858999 10065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:28:06.860208 10065 hybrid_clock.cc:648] HybridClock initialized: now 1790544486860161 us; error 49 us; skew 500 ppm
I20260927 21:28:06.860533 10065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:28:06.862823 10065 webserver.cc:460] Webserver started at http://127.0.97.3:43169/ using document root <none> and password file <none>
I20260927 21:28:06.863525 10065 fs_manager.cc:359] Metadata directory not provided
I20260927 21:28:06.863613 10065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:28:06.863965 10065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:28:06.866950 10065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/data/instance:
uuid: "ee8c3dfdfc3c40b6b88c4a5e629e23e3"
format_stamp: "Formatted at 2026-09-27 21:28:06 on dist-test-slave-bxm0"
server_key: "866533531edd009abb8f6aed0abdc32e"
server_key_iv: "e4f544d870b6f197fb6e712564b76dc6"
server_key_version: "encryptionkey@0"
I20260927 21:28:06.867779 10065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ee8c3dfdfc3c40b6b88c4a5e629e23e3"
format_stamp: "Formatted at 2026-09-27 21:28:06 on dist-test-slave-bxm0"
server_key: "866533531edd009abb8f6aed0abdc32e"
server_key_iv: "e4f544d870b6f197fb6e712564b76dc6"
server_key_version: "encryptionkey@0"
I20260927 21:28:06.873386 10065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:28:06.877012 10079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:06.878500 10065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:28:06.878695 10065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "ee8c3dfdfc3c40b6b88c4a5e629e23e3"
format_stamp: "Formatted at 2026-09-27 21:28:06 on dist-test-slave-bxm0"
server_key: "866533531edd009abb8f6aed0abdc32e"
server_key_iv: "e4f544d870b6f197fb6e712564b76dc6"
server_key_version: "encryptionkey@0"
I20260927 21:28:06.878948 10065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:28:06.911619 10065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:28:06.912537 10065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:28:06.912824 10065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:28:06.913928 10065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:28:06.915513 10065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:28:06.915638 10065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:06.915769 10065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:28:06.915866 10065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:06.949767 10065 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:32857
I20260927 21:28:06.949872 10191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:32857 every 8 connection(s)
I20260927 21:28:06.951076 10065 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:28:06.954350 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 10065
I20260927 21:28:06.954499 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/wal/instance
I20260927 21:28:06.955123 388 external_mini_cluster.cc:1428] Setting key ac4f197934f72ab091a540c72097e904
I20260927 21:28:06.964092 9933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:06.953788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45509 (local address 127.0.97.62:40069)
0927 21:28:06.953973 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:06.953978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:06.955665 (+ 1687us) server_negotiation.cc:407] Connection header received
0927 21:28:06.957953 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:06.957960 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:06.958056 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:06.958173 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:06.959193 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:06.959949 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:06.961458 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:06.961982 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:06.962100 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:06.963394 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:28:06.963399 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:28:06.963402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:28:06.963420 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:28:06.963450 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:28:06.963462 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:28:06.963544 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:06.963741 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 21:28:06.963822 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 21:28:06.965029 10194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:06.954321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40069 (local address 127.0.97.3:45509)
0927 21:28:06.955491 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0927 21:28:06.955546 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 21:28:06.957636 (+ 2090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:28:06.958373 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:28:06.958381 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:28:06.958469 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:28:06.959021 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:06.959037 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:06.960077 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:06.960080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:06.961323 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:06.961333 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:06.962170 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:06.962174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:06.962379 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:06.963103 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:28:06.963106 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:28:06.963144 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:28:06.963219 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:28:06.963552 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:28:06.963559 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:28:06.963564 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:28:06.963701 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 21:28:06.963987 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":348,"threads_started":1}
I20260927 21:28:06.966511 10192 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40069
I20260927 21:28:06.966796 10192 heartbeater.cc:464] Registering TS with master...
I20260927 21:28:06.967554 10192 heartbeater.cc:511] Master 127.0.97.62:40069 requested a full tablet report, sending...
I20260927 21:28:06.969054 9745 ts_manager.cc:194] Registered new tserver with Master: ee8c3dfdfc3c40b6b88c4a5e629e23e3 (127.0.97.3:32857)
I20260927 21:28:06.970165 9745 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:45509
I20260927 21:28:06.970809 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 21:28:06.977772 9933 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:28:06.976183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51382 (local address 127.0.97.62:40069)
0927 21:28:06.976396 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:06.976400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:06.976869 (+ 469us) server_negotiation.cc:407] Connection header received
0927 21:28:06.976997 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:06.977001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:06.977078 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:06.977166 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:06.977499 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51382: BlockingRecv error: recv got EOF from 127.0.0.1:51382 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 21:28:06.978366 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 9804
I20260927 21:28:06.986207 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:06.979232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44142 (local address 127.0.97.1:40357)
0927 21:28:06.979738 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:06.979747 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:06.979788 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:28:06.979841 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:06.979847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:06.980054 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:06.980179 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:06.980537 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:06.981503 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:06.982930 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:06.983543 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:06.983738 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:06.984747 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:28:06.984755 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:28:06.984785 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:28:06.985398 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:28:06.985442 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:28:06.985460 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:28:06.985554 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:06.985807 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 21:28:06.985926 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":261,"threads_started":1}
I20260927 21:28:06.987421 9926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:28:07.068830 9931 heartbeater.cc:503] Master 127.0.97.62:40069 was elected leader, sending a full tablet report...
I20260927 21:28:07.070667 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 9804
I20260927 21:28:07.087162 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 9935
I20260927 21:28:07.095206 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:07.088181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43116 (local address 127.0.97.2:39329)
0927 21:28:07.088687 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:07.088695 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:07.088729 (+ 34us) server_negotiation.cc:407] Connection header received
0927 21:28:07.088788 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:07.088795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:07.089009 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:07.089187 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:07.089595 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:07.090635 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:07.091882 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:07.092463 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:07.092606 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:07.093695 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:28:07.093750 (+ 55us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:28:07.093764 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:28:07.094368 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:28:07.094420 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:28:07.094439 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:28:07.094568 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:07.094734 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 21:28:07.094881 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":252,"threads_started":1}
I20260927 21:28:07.096514 10057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:28:07.178085 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 9935
I20260927 21:28:07.194329 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 10065
I20260927 21:28:07.202833 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:07.195431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46632 (local address 127.0.97.3:32857)
0927 21:28:07.196027 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:07.196034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:07.196064 (+ 30us) server_negotiation.cc:407] Connection header received
0927 21:28:07.196123 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:07.196129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:07.196373 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:07.196566 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:07.197034 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:07.198129 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:07.199388 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:07.199947 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:07.200086 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:07.201186 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:28:07.201194 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:28:07.201207 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:28:07.201968 (+ 761us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:28:07.202027 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:28:07.202045 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:28:07.202173 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:07.202321 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 21:28:07.202454 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":308,"threads_started":1}
I20260927 21:28:07.204170 10187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:28:07.287014 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 10065
I20260927 21:28:07.302897 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 9714
I20260927 21:28:07.304136 9775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:28:07.408078 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 9714
2026-09-27T21:28:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2466 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T21:28:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:28:07Z Disabled control of system clock
I20260927 21:28:07.480253 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35781
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 21:28:07.815403 10211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:28:07.815701 10211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:28:07.815903 10211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:28:07.823287 10211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:28:07.823400 10211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:28:07.823513 10211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:28:07.823617 10211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:28:07.823681 10211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:28:07.833084 10211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35781
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:28:07.834504 10211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:28:07.836272 10211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:28:07.848177 10216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:28:07.848134 10219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:28:07.848750 10217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:28:07.848873 10211 server_base.cc:1034] running on GCE node
I20260927 21:28:07.849817 10211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:28:07.851221 10211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:28:07.852448 10211 hybrid_clock.cc:648] HybridClock initialized: now 1790544487852406 us; error 39 us; skew 500 ppm
I20260927 21:28:07.852823 10211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:28:07.855249 10211 webserver.cc:460] Webserver started at http://127.0.97.62:39029/ using document root <none> and password file <none>
I20260927 21:28:07.855970 10211 fs_manager.cc:359] Metadata directory not provided
I20260927 21:28:07.856088 10211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:28:07.856427 10211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:28:07.859481 10211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "c0ed4e7621c643ad88b773505fd88182"
format_stamp: "Formatted at 2026-09-27 21:28:07 on dist-test-slave-bxm0"
server_key: "271f9c4a16be82d2880eb13443360a13"
server_key_iv: "a8bdadac2270c1d3998d986bb9588c6f"
server_key_version: "encryptionkey@0"
I20260927 21:28:07.860306 10211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "c0ed4e7621c643ad88b773505fd88182"
format_stamp: "Formatted at 2026-09-27 21:28:07 on dist-test-slave-bxm0"
server_key: "271f9c4a16be82d2880eb13443360a13"
server_key_iv: "a8bdadac2270c1d3998d986bb9588c6f"
server_key_version: "encryptionkey@0"
I20260927 21:28:07.865453 10211 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 21:28:07.868880 10225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:07.870302 10211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:28:07.870523 10211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "c0ed4e7621c643ad88b773505fd88182"
format_stamp: "Formatted at 2026-09-27 21:28:07 on dist-test-slave-bxm0"
server_key: "271f9c4a16be82d2880eb13443360a13"
server_key_iv: "a8bdadac2270c1d3998d986bb9588c6f"
server_key_version: "encryptionkey@0"
I20260927 21:28:07.870751 10211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:28:07.877676 10211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:28:07.878547 10211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:28:07.878868 10211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:28:07.898152 10211 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35781
I20260927 21:28:07.898177 10276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35781 every 8 connection(s)
I20260927 21:28:07.899634 10211 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:28:07.902797 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 10211
I20260927 21:28:07.903051 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:28:07.903617 10277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:28:07.903687 388 external_mini_cluster.cc:1428] Setting key 0d35b6603c94a8f8a2249b1e691c2039
I20260927 21:28:07.912966 10280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:07.905763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36804 (local address 127.0.97.62:35781)
0927 21:28:07.906914 (+ 1151us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:07.906929 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:07.906978 (+ 49us) server_negotiation.cc:407] Connection header received
0927 21:28:07.908735 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:07.908765 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:07.909375 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:07.909846 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:07.910893 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:28:07.910901 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:28:07.910917 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:28:07.910939 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:28:07.911026 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:28:07.911049 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:28:07.911285 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:07.911425 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 21:28:07.911708 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":385,"threads_started":1}
I20260927 21:28:07.918740 10277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0ed4e7621c643ad88b773505fd88182: Bootstrap starting.
I20260927 21:28:07.921832 10277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0ed4e7621c643ad88b773505fd88182: Neither blocks nor log segments found. Creating new log.
I20260927 21:28:07.923087 10277 log.cc:824] T 00000000000000000000000000000000 P c0ed4e7621c643ad88b773505fd88182: Log is configured to *not* fsync() on all Append() calls
I20260927 21:28:07.925896 10277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0ed4e7621c643ad88b773505fd88182: No bootstrap required, opened a new log
I20260927 21:28:07.933458 10277 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0ed4e7621c643ad88b773505fd88182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ed4e7621c643ad88b773505fd88182" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35781 } }
I20260927 21:28:07.933923 10277 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0ed4e7621c643ad88b773505fd88182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:28:07.934067 10277 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0ed4e7621c643ad88b773505fd88182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ed4e7621c643ad88b773505fd88182, State: Initialized, Role: FOLLOWER
I20260927 21:28:07.934660 10277 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0ed4e7621c643ad88b773505fd88182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ed4e7621c643ad88b773505fd88182" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35781 } }
I20260927 21:28:07.934900 10277 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0ed4e7621c643ad88b773505fd88182 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:28:07.935032 10277 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0ed4e7621c643ad88b773505fd88182 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:28:07.935200 10277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0ed4e7621c643ad88b773505fd88182 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:28:07.937333 10277 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0ed4e7621c643ad88b773505fd88182 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ed4e7621c643ad88b773505fd88182" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35781 } }
I20260927 21:28:07.937911 10277 leader_election.cc:302] T 00000000000000000000000000000000 P c0ed4e7621c643ad88b773505fd88182 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0ed4e7621c643ad88b773505fd88182; no voters:
I20260927 21:28:07.938510 10277 leader_election.cc:288] T 00000000000000000000000000000000 P c0ed4e7621c643ad88b773505fd88182 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:28:07.938756 10282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0ed4e7621c643ad88b773505fd88182 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:28:07.939510 10282 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0ed4e7621c643ad88b773505fd88182 [term 1 LEADER]: Becoming Leader. State: Replica: c0ed4e7621c643ad88b773505fd88182, State: Running, Role: LEADER
I20260927 21:28:07.940093 10282 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0ed4e7621c643ad88b773505fd88182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ed4e7621c643ad88b773505fd88182" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35781 } }
I20260927 21:28:07.940725 10277 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0ed4e7621c643ad88b773505fd88182 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:28:07.942920 10284 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0ed4e7621c643ad88b773505fd88182 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0ed4e7621c643ad88b773505fd88182. Latest consensus state: current_term: 1 leader_uuid: "c0ed4e7621c643ad88b773505fd88182" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ed4e7621c643ad88b773505fd88182" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35781 } } }
I20260927 21:28:07.943276 10284 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0ed4e7621c643ad88b773505fd88182 [sys.catalog]: This master's current role is: LEADER
I20260927 21:28:07.944574 10283 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0ed4e7621c643ad88b773505fd88182 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0ed4e7621c643ad88b773505fd88182" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ed4e7621c643ad88b773505fd88182" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35781 } } }
I20260927 21:28:07.945125 10283 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0ed4e7621c643ad88b773505fd88182 [sys.catalog]: This master's current role is: LEADER
I20260927 21:28:07.946118 10287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:28:07.952382 10287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:28:07.961429 10287 catalog_manager.cc:1347] Generated new cluster ID: 671c5f7bf8c14d0d942b27d3593b8c65
I20260927 21:28:07.961556 10287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:28:07.994026 10287 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:28:07.994581 10287 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:28:08.003692 10287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0ed4e7621c643ad88b773505fd88182: Generated new TSK 0
I20260927 21:28:08.004796 10287 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:28:08.014096 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35781
--builtin_ntp_servers=127.0.97.20:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 21:28:08.359256 10301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:28:08.359606 10301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:28:08.359776 10301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:28:08.367183 10301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:28:08.367347 10301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:28:08.367534 10301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:28:08.377126 10301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:28:08.378607 10301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:28:08.380379 10301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:28:08.393540 10306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:28:08.393630 10307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:28:08.393508 10309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:28:08.399334 10301 server_base.cc:1034] running on GCE node
I20260927 21:28:08.400046 10301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:28:08.401106 10301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:28:08.402354 10301 hybrid_clock.cc:648] HybridClock initialized: now 1790544488402299 us; error 52 us; skew 500 ppm
I20260927 21:28:08.402699 10301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:28:08.405074 10301 webserver.cc:460] Webserver started at http://127.0.97.1:39927/ using document root <none> and password file <none>
I20260927 21:28:08.405869 10301 fs_manager.cc:359] Metadata directory not provided
I20260927 21:28:08.405995 10301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:28:08.406319 10301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:28:08.410938 10301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/data/instance:
uuid: "5259e285d68f4fb38d26cc5499a2fd46"
format_stamp: "Formatted at 2026-09-27 21:28:08 on dist-test-slave-bxm0"
server_key: "59e73f26c0fb13f4aec6049f6c739a14"
server_key_iv: "7e935ad66c2afbe496ba01838d144c73"
server_key_version: "encryptionkey@0"
I20260927 21:28:08.411755 10301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5259e285d68f4fb38d26cc5499a2fd46"
format_stamp: "Formatted at 2026-09-27 21:28:08 on dist-test-slave-bxm0"
server_key: "59e73f26c0fb13f4aec6049f6c739a14"
server_key_iv: "7e935ad66c2afbe496ba01838d144c73"
server_key_version: "encryptionkey@0"
I20260927 21:28:08.417323 10301 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 21:28:08.421478 10315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:08.423084 10301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:28:08.423314 10301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "5259e285d68f4fb38d26cc5499a2fd46"
format_stamp: "Formatted at 2026-09-27 21:28:08 on dist-test-slave-bxm0"
server_key: "59e73f26c0fb13f4aec6049f6c739a14"
server_key_iv: "7e935ad66c2afbe496ba01838d144c73"
server_key_version: "encryptionkey@0"
I20260927 21:28:08.423579 10301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:28:08.429790 10301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:28:08.430663 10301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:28:08.430946 10301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:28:08.431891 10301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:28:08.433362 10301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:28:08.433506 10301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:08.433719 10301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:28:08.433822 10301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 21:28:08.468207 10301 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40903
I20260927 21:28:08.468255 10427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40903 every 8 connection(s)
I20260927 21:28:08.469568 10301 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:28:08.470023 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 10301
I20260927 21:28:08.470227 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-0/wal/instance
I20260927 21:28:08.471084 388 external_mini_cluster.cc:1428] Setting key 73cd150cead139de84ec2eb54659b03e
I20260927 21:28:08.474242 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35781
--builtin_ntp_servers=127.0.97.20:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 21:28:08.478833 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:08.472306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34881 (local address 127.0.97.62:35781)
0927 21:28:08.472792 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:08.472806 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:08.474427 (+ 1621us) server_negotiation.cc:407] Connection header received
0927 21:28:08.476176 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:08.476181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:08.476255 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:08.476387 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:08.478043 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:28:08.478049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:28:08.478054 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:28:08.478075 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:28:08.478125 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:28:08.478139 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:28:08.478276 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:08.478464 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 21:28:08.478533 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":259,"threads_started":1}
I20260927 21:28:08.479792 10431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:08.473099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35781 (local address 127.0.97.1:34881)
0927 21:28:08.474229 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0927 21:28:08.474280 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 21:28:08.475904 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:28:08.476776 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:28:08.476786 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:28:08.476910 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:28:08.477737 (+ 827us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:28:08.477739 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:28:08.477777 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:28:08.477866 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:28:08.478235 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:28:08.478245 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:28:08.478253 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 21:28:08.478549 (+ 296us) client_negotiation.cc:240] Negotiation successful
0927 21:28:08.478815 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":328,"threads_started":1}
I20260927 21:28:08.481478 10428 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35781
I20260927 21:28:08.481873 10428 heartbeater.cc:464] Registering TS with master...
I20260927 21:28:08.482492 10428 heartbeater.cc:511] Master 127.0.97.62:35781 requested a full tablet report, sending...
I20260927 21:28:08.484620 10242 ts_manager.cc:194] Registered new tserver with Master: 5259e285d68f4fb38d26cc5499a2fd46 (127.0.97.1:40903)
W20260927 21:28:08.823220 10432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:28:08.823560 10432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:28:08.823715 10432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:28:08.832050 10432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:28:08.832207 10432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:28:08.832378 10432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:28:08.842521 10432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:28:08.843855 10432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:28:08.845675 10432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:28:08.859540 10440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:28:08.859725 10438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:28:08.859671 10437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:28:08.860648 10432 server_base.cc:1034] running on GCE node
I20260927 21:28:08.861344 10432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:28:08.862417 10432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:28:08.863653 10432 hybrid_clock.cc:648] HybridClock initialized: now 1790544488863592 us; error 56 us; skew 500 ppm
I20260927 21:28:08.864018 10432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:28:08.866591 10432 webserver.cc:460] Webserver started at http://127.0.97.2:46475/ using document root <none> and password file <none>
I20260927 21:28:08.867305 10432 fs_manager.cc:359] Metadata directory not provided
I20260927 21:28:08.867439 10432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:28:08.867771 10432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:28:08.870760 10432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/data/instance:
uuid: "b0949ced68784a53bdec64a05964881f"
format_stamp: "Formatted at 2026-09-27 21:28:08 on dist-test-slave-bxm0"
server_key: "acae8c598046cf8f8ce2d5deeb5b30d8"
server_key_iv: "a94908fd6223b1c9c6bd4877e1d0a396"
server_key_version: "encryptionkey@0"
I20260927 21:28:08.871541 10432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b0949ced68784a53bdec64a05964881f"
format_stamp: "Formatted at 2026-09-27 21:28:08 on dist-test-slave-bxm0"
server_key: "acae8c598046cf8f8ce2d5deeb5b30d8"
server_key_iv: "a94908fd6223b1c9c6bd4877e1d0a396"
server_key_version: "encryptionkey@0"
I20260927 21:28:08.877267 10432 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:28:08.880997 10446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:08.882493 10432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:28:08.882699 10432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "b0949ced68784a53bdec64a05964881f"
format_stamp: "Formatted at 2026-09-27 21:28:08 on dist-test-slave-bxm0"
server_key: "acae8c598046cf8f8ce2d5deeb5b30d8"
server_key_iv: "a94908fd6223b1c9c6bd4877e1d0a396"
server_key_version: "encryptionkey@0"
I20260927 21:28:08.882969 10432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:28:08.888998 10432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:28:08.889979 10432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:28:08.890267 10432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:28:08.891237 10432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:28:08.892689 10432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:28:08.892825 10432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:08.892968 10432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:28:08.893056 10432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:08.928886 10432 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37839
I20260927 21:28:08.928977 10558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37839 every 8 connection(s)
I20260927 21:28:08.930321 10432 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:28:08.937808 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 10432
I20260927 21:28:08.937991 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-1/wal/instance
I20260927 21:28:08.938412 388 external_mini_cluster.cc:1428] Setting key 8684a673aa6ce5a5a6c8fff4c1711af2
I20260927 21:28:08.938896 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:08.933008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43773 (local address 127.0.97.62:35781)
0927 21:28:08.933198 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:08.933203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:08.934658 (+ 1455us) server_negotiation.cc:407] Connection header received
0927 21:28:08.936345 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:08.936349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:08.936405 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:08.936498 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:08.938103 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:28:08.938110 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:28:08.938114 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:28:08.938136 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:28:08.938182 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:28:08.938195 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:28:08.938332 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:08.938530 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 21:28:08.938602 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 21:28:08.939903 10561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:08.933534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35781 (local address 127.0.97.2:43773)
0927 21:28:08.934496 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0927 21:28:08.934543 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:28:08.936059 (+ 1516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:28:08.936794 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:28:08.936803 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:28:08.936901 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:28:08.937729 (+ 828us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:28:08.937732 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:28:08.937773 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:28:08.937866 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:28:08.938300 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:28:08.938310 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:28:08.938317 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 21:28:08.938616 (+ 299us) client_negotiation.cc:240] Negotiation successful
0927 21:28:08.938897 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":270,"threads_started":1}
I20260927 21:28:08.941470 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35781
--builtin_ntp_servers=127.0.97.20:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 21:28:08.941501 10559 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35781
I20260927 21:28:08.941900 10559 heartbeater.cc:464] Registering TS with master...
I20260927 21:28:08.942687 10559 heartbeater.cc:511] Master 127.0.97.62:35781 requested a full tablet report, sending...
I20260927 21:28:08.944466 10242 ts_manager.cc:194] Registered new tserver with Master: b0949ced68784a53bdec64a05964881f (127.0.97.2:37839)
W20260927 21:28:09.274063 10562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:28:09.274348 10562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:28:09.274434 10562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:28:09.281594 10562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:28:09.281795 10562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:28:09.281909 10562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:28:09.291383 10562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:28:09.292623 10562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:28:09.294312 10562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:28:09.307472 10567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:28:09.307367 10568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:28:09.307376 10570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:28:09.308139 10562 server_base.cc:1034] running on GCE node
I20260927 21:28:09.308784 10562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:28:09.309742 10562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:28:09.310961 10562 hybrid_clock.cc:648] HybridClock initialized: now 1790544489310911 us; error 37 us; skew 500 ppm
I20260927 21:28:09.311329 10562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:28:09.313845 10562 webserver.cc:460] Webserver started at http://127.0.97.3:36989/ using document root <none> and password file <none>
I20260927 21:28:09.314574 10562 fs_manager.cc:359] Metadata directory not provided
I20260927 21:28:09.314699 10562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:28:09.315034 10562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:28:09.318030 10562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/data/instance:
uuid: "3303ee1c5c1f4d8c805a4fe76005bebe"
format_stamp: "Formatted at 2026-09-27 21:28:09 on dist-test-slave-bxm0"
server_key: "ae0c877116137886f18d61c536ef0202"
server_key_iv: "01b38530d4236d4d2846bafd19b6b7d5"
server_key_version: "encryptionkey@0"
I20260927 21:28:09.318841 10562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3303ee1c5c1f4d8c805a4fe76005bebe"
format_stamp: "Formatted at 2026-09-27 21:28:09 on dist-test-slave-bxm0"
server_key: "ae0c877116137886f18d61c536ef0202"
server_key_iv: "01b38530d4236d4d2846bafd19b6b7d5"
server_key_version: "encryptionkey@0"
I20260927 21:28:09.324333 10562 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:28:09.328044 10576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:09.329519 10562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:28:09.329764 10562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "3303ee1c5c1f4d8c805a4fe76005bebe"
format_stamp: "Formatted at 2026-09-27 21:28:09 on dist-test-slave-bxm0"
server_key: "ae0c877116137886f18d61c536ef0202"
server_key_iv: "01b38530d4236d4d2846bafd19b6b7d5"
server_key_version: "encryptionkey@0"
I20260927 21:28:09.330009 10562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:28:09.335770 10562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:28:09.336585 10562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:28:09.336864 10562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:28:09.337870 10562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:28:09.339334 10562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:28:09.339447 10562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:09.339577 10562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:28:09.339669 10562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:09.373013 10562 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44709
I20260927 21:28:09.373104 10688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44709 every 8 connection(s)
I20260927 21:28:09.374394 10562 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:28:09.375293 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 10562
I20260927 21:28:09.375473 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790544390748963-388-0/minicluster-data/ts-2/wal/instance
I20260927 21:28:09.376011 388 external_mini_cluster.cc:1428] Setting key 8426ad5b3c3952acdba74bef1cc52828
I20260927 21:28:09.383189 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:09.377225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58121 (local address 127.0.97.62:35781)
0927 21:28:09.377406 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:09.377412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:09.379138 (+ 1726us) server_negotiation.cc:407] Connection header received
0927 21:28:09.380913 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:09.380917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:09.380990 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:09.381108 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:09.382463 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:28:09.382469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:28:09.382473 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:28:09.382491 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:28:09.382531 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:28:09.382544 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:28:09.382669 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:09.382854 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 21:28:09.382920 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:28:09.384145 10691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:09.377981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35781 (local address 127.0.97.3:58121)
0927 21:28:09.378979 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0927 21:28:09.379051 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 21:28:09.380654 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:28:09.381344 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:28:09.381352 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:28:09.381447 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:28:09.382169 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:28:09.382171 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:28:09.382210 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:28:09.382296 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:28:09.382635 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:28:09.382646 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:28:09.382653 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 21:28:09.382946 (+ 293us) client_negotiation.cc:240] Negotiation successful
0927 21:28:09.383211 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":306,"threads_started":1}
I20260927 21:28:09.385462 10689 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35781
I20260927 21:28:09.385823 10689 heartbeater.cc:464] Registering TS with master...
I20260927 21:28:09.386430 10689 heartbeater.cc:511] Master 127.0.97.62:35781 requested a full tablet report, sending...
I20260927 21:28:09.387883 10242 ts_manager.cc:194] Registered new tserver with Master: 3303ee1c5c1f4d8c805a4fe76005bebe (127.0.97.3:44709)
I20260927 21:28:09.391778 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 21:28:09.397891 10430 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:28:09.396269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36816 (local address 127.0.97.62:35781)
0927 21:28:09.396431 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:09.396436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:09.396910 (+ 474us) server_negotiation.cc:407] Connection header received
0927 21:28:09.397118 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:09.397122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:09.397174 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:09.397272 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:09.397671 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36816: BlockingRecv error: recv got EOF from 127.0.0.1:36816 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:28:09.398278 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 10301
I20260927 21:28:09.401773 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:09.399257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49972 (local address 127.0.97.1:40903)
0927 21:28:09.399800 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:09.399807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:09.399832 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:28:09.399888 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:09.399894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:09.400109 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:09.400232 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:09.401108 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:28:09.401115 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:28:09.401125 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:28:09.401138 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:28:09.401185 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:28:09.401199 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:28:09.401297 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:09.401371 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 21:28:09.401448 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":273,"threads_started":1}
I20260927 21:28:09.403000 10423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:28:09.485136 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 10301
I20260927 21:28:09.501478 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 10432
I20260927 21:28:09.505436 10700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:09.502574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36550 (local address 127.0.97.2:37839)
0927 21:28:09.503144 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:09.503151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:09.503177 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:28:09.503243 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:09.503249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:09.503464 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:09.503591 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:09.504559 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:28:09.504566 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:28:09.504578 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:28:09.504590 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:28:09.504632 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:28:09.504646 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:28:09.504750 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:09.504997 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 21:28:09.505145 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":301,"threads_started":1}
I20260927 21:28:09.506875 10554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:28:09.587566 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 10432
I20260927 21:28:09.604347 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 10562
I20260927 21:28:09.608399 10702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:09.605447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47638 (local address 127.0.97.3:44709)
0927 21:28:09.606222 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:09.606229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:09.606260 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:28:09.606319 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:09.606325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:09.606571 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:09.606708 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:09.607640 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:28:09.607648 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:28:09.607662 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:28:09.607677 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:28:09.607714 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:28:09.607727 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:28:09.607856 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:09.607916 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 21:28:09.608026 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":454,"threads_started":1}
I20260927 21:28:09.609738 10684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:28:09.694963 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 10562
I20260927 21:28:09.710767 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 10211
I20260927 21:28:09.711966 10272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:28:09.812707 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 10211
2026-09-27T21:28:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2405 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:28:09 dist-test-slave-bxm0 krb5kdc[10707](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:28:09 dist-test-slave-bxm0 krb5kdc[10707](info): set up 2 sockets
Sep 27 21:28:09 dist-test-slave-bxm0 krb5kdc[10707](info): commencing operation
krb5kdc: starting...
W20260927 21:28:10.914598 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:28:11 dist-test-slave-bxm0 krb5kdc[10707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:28:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:28:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:28:11.153285 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42553
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:28:11.483948 10720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:28:11.484331 10720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:28:11.484493 10720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:28:11.491887 10720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:28:11.492057 10720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:28:11.492165 10720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:28:11.492271 10720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:28:11.492345 10720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:28:11.502518 10720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:28:11.503952 10720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:28:11.505342 10720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:28:11.517509 10725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:28:11.517539 10728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:28:11.517427 10726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:28:11.518530 10720 server_base.cc:1034] running on GCE node
I20260927 21:28:11.519449 10720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:28:11.520942 10720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:28:11.522161 10720 hybrid_clock.cc:648] HybridClock initialized: now 1790544491522111 us; error 49 us; skew 500 ppm
I20260927 21:28:11.522533 10720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:28:11 dist-test-slave-bxm0 krb5kdc[10707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:28:11.529670 10720 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:28:11.531392 10720 webserver.cc:460] Webserver started at http://127.0.97.62:34739/ using document root <none> and password file <none>
I20260927 21:28:11.532126 10720 fs_manager.cc:359] Metadata directory not provided
I20260927 21:28:11.532277 10720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:28:11.532598 10720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:28:11.535636 10720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "d5556e1d5c9f4aee9a4b16eb4c62fd02"
format_stamp: "Formatted at 2026-09-27 21:28:11 on dist-test-slave-bxm0"
server_key: "c944f654228fa3f964316a40974574db"
server_key_iv: "f1852d96f2cd745094aaea0aedefcb90"
server_key_version: "encryptionkey@0"
I20260927 21:28:11.536450 10720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "d5556e1d5c9f4aee9a4b16eb4c62fd02"
format_stamp: "Formatted at 2026-09-27 21:28:11 on dist-test-slave-bxm0"
server_key: "c944f654228fa3f964316a40974574db"
server_key_iv: "f1852d96f2cd745094aaea0aedefcb90"
server_key_version: "encryptionkey@0"
I20260927 21:28:11.541712 10720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 21:28:11.545168 10735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:11.546590 10720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:28:11.546777 10720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "d5556e1d5c9f4aee9a4b16eb4c62fd02"
format_stamp: "Formatted at 2026-09-27 21:28:11 on dist-test-slave-bxm0"
server_key: "c944f654228fa3f964316a40974574db"
server_key_iv: "f1852d96f2cd745094aaea0aedefcb90"
server_key_version: "encryptionkey@0"
I20260927 21:28:11.546974 10720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:28:11.563830 10720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:28:11.567965 10720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:28:11.568259 10720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:28:11.586786 10720 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42553
I20260927 21:28:11.586830 10786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42553 every 8 connection(s)
I20260927 21:28:11.588217 10720 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:28:11.592110 10787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:28:11.597947 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 10720
I20260927 21:28:11.598165 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:28:11.598784 388 external_mini_cluster.cc:1428] Setting key e36edc7e08a589d34e1b406abd6f5ef1
I20260927 21:28:11.603259 10787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5556e1d5c9f4aee9a4b16eb4c62fd02: Bootstrap starting.
I20260927 21:28:11.607062 10787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5556e1d5c9f4aee9a4b16eb4c62fd02: Neither blocks nor log segments found. Creating new log.
I20260927 21:28:11.608336 10787 log.cc:824] T 00000000000000000000000000000000 P d5556e1d5c9f4aee9a4b16eb4c62fd02: Log is configured to *not* fsync() on all Append() calls
I20260927 21:28:11.611763 10787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5556e1d5c9f4aee9a4b16eb4c62fd02: No bootstrap required, opened a new log
Sep 27 21:28:11 dist-test-slave-bxm0 krb5kdc[10707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:28:11.622028 10787 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5556e1d5c9f4aee9a4b16eb4c62fd02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5556e1d5c9f4aee9a4b16eb4c62fd02" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42553 } }
I20260927 21:28:11.622586 10787 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5556e1d5c9f4aee9a4b16eb4c62fd02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:28:11.622705 10787 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5556e1d5c9f4aee9a4b16eb4c62fd02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5556e1d5c9f4aee9a4b16eb4c62fd02, State: Initialized, Role: FOLLOWER
I20260927 21:28:11.623469 10787 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5556e1d5c9f4aee9a4b16eb4c62fd02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5556e1d5c9f4aee9a4b16eb4c62fd02" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42553 } }
I20260927 21:28:11.623773 10787 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5556e1d5c9f4aee9a4b16eb4c62fd02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:28:11.623942 10787 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5556e1d5c9f4aee9a4b16eb4c62fd02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:28:11.624126 10787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5556e1d5c9f4aee9a4b16eb4c62fd02 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:28:11.624518 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:11.600931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36640 (local address 127.0.97.62:42553)
0927 21:28:11.601796 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:11.601809 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:11.601853 (+ 44us) server_negotiation.cc:407] Connection header received
0927 21:28:11.603553 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:11.603575 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:11.603977 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:11.604307 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:11.606173 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:11.607522 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:11.609314 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:11.610418 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:11.610598 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:11.614538 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:28:11.614548 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:28:11.614571 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:28:11.614624 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:28:11.618758 (+ 4134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:28:11.619595 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:28:11.619605 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:28:11.619615 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:28:11.619865 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:28:11.620251 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:28:11.620256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:28:11.620260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:28:11.621075 (+ 815us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:28:11.621346 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:11.622135 (+ 789us) server_negotiation.cc:299] Negotiation successful
0927 21:28:11.622999 (+ 864us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":396,"threads_started":1}
I20260927 21:28:11.627290 10787 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5556e1d5c9f4aee9a4b16eb4c62fd02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5556e1d5c9f4aee9a4b16eb4c62fd02" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42553 } }
I20260927 21:28:11.627952 10787 leader_election.cc:302] T 00000000000000000000000000000000 P d5556e1d5c9f4aee9a4b16eb4c62fd02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5556e1d5c9f4aee9a4b16eb4c62fd02; no voters:
I20260927 21:28:11.628705 10787 leader_election.cc:288] T 00000000000000000000000000000000 P d5556e1d5c9f4aee9a4b16eb4c62fd02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:28:11.628886 10792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5556e1d5c9f4aee9a4b16eb4c62fd02 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:28:11.630230 10792 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5556e1d5c9f4aee9a4b16eb4c62fd02 [term 1 LEADER]: Becoming Leader. State: Replica: d5556e1d5c9f4aee9a4b16eb4c62fd02, State: Running, Role: LEADER
I20260927 21:28:11.630832 10792 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5556e1d5c9f4aee9a4b16eb4c62fd02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5556e1d5c9f4aee9a4b16eb4c62fd02" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42553 } }
I20260927 21:28:11.631170 10787 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5556e1d5c9f4aee9a4b16eb4c62fd02 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:28:11.633697 10794 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5556e1d5c9f4aee9a4b16eb4c62fd02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5556e1d5c9f4aee9a4b16eb4c62fd02. Latest consensus state: current_term: 1 leader_uuid: "d5556e1d5c9f4aee9a4b16eb4c62fd02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5556e1d5c9f4aee9a4b16eb4c62fd02" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42553 } } }
I20260927 21:28:11.634133 10794 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5556e1d5c9f4aee9a4b16eb4c62fd02 [sys.catalog]: This master's current role is: LEADER
I20260927 21:28:11.633755 10793 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5556e1d5c9f4aee9a4b16eb4c62fd02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5556e1d5c9f4aee9a4b16eb4c62fd02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5556e1d5c9f4aee9a4b16eb4c62fd02" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42553 } } }
I20260927 21:28:11.634272 10793 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5556e1d5c9f4aee9a4b16eb4c62fd02 [sys.catalog]: This master's current role is: LEADER
I20260927 21:28:11.636011 10799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:28:11.641445 10799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:28:11.650441 10799 catalog_manager.cc:1347] Generated new cluster ID: cca583155f28413e92ec3b8972e958fc
I20260927 21:28:11.650566 10799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:28:11.672739 10799 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:28:11.674157 10799 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:28:11.686195 10799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5556e1d5c9f4aee9a4b16eb4c62fd02: Generated new TSK 0
I20260927 21:28:11.687096 10799 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 21:28:11.747622 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42553
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:28:12.086452 10813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:28:12.086769 10813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:28:12.086934 10813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:28:12.094630 10813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:28:12.094769 10813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:28:12.094960 10813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:28:12.105028 10813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:28:12.106491 10813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:28:12.107841 10813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:28:12.121337 10818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:28:12.121389 10819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:28:12.121358 10821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:28:12.122303 10813 server_base.cc:1034] running on GCE node
I20260927 21:28:12.122977 10813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:28:12.123924 10813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:28:12.125150 10813 hybrid_clock.cc:648] HybridClock initialized: now 1790544492125101 us; error 45 us; skew 500 ppm
I20260927 21:28:12.125464 10813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:28:12 dist-test-slave-bxm0 krb5kdc[10707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:28:12.128858 10813 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 21:28:12.130378 10813 webserver.cc:460] Webserver started at http://127.0.97.1:44563/ using document root <none> and password file <none>
I20260927 21:28:12.131120 10813 fs_manager.cc:359] Metadata directory not provided
I20260927 21:28:12.131268 10813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:28:12.131580 10813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:28:12.134745 10813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-0/data/instance:
uuid: "2e0fb4e8f94141488fddfb40d3e11f4d"
format_stamp: "Formatted at 2026-09-27 21:28:12 on dist-test-slave-bxm0"
server_key: "40313dc58e552cb359c885dbc530d113"
server_key_iv: "f4f4aa9abbd36b20255b099ae5542df0"
server_key_version: "encryptionkey@0"
I20260927 21:28:12.135560 10813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2e0fb4e8f94141488fddfb40d3e11f4d"
format_stamp: "Formatted at 2026-09-27 21:28:12 on dist-test-slave-bxm0"
server_key: "40313dc58e552cb359c885dbc530d113"
server_key_iv: "f4f4aa9abbd36b20255b099ae5542df0"
server_key_version: "encryptionkey@0"
I20260927 21:28:12.140933 10813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:28:12.144433 10828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:12.145875 10813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:28:12.146049 10813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "2e0fb4e8f94141488fddfb40d3e11f4d"
format_stamp: "Formatted at 2026-09-27 21:28:12 on dist-test-slave-bxm0"
server_key: "40313dc58e552cb359c885dbc530d113"
server_key_iv: "f4f4aa9abbd36b20255b099ae5542df0"
server_key_version: "encryptionkey@0"
I20260927 21:28:12.146253 10813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:28:12.165947 10813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:28:12.170476 10813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:28:12.170849 10813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:28:12.171885 10813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:28:12.173321 10813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:28:12.173422 10813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:12.173576 10813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:28:12.173672 10813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 21:28:12.206991 10813 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43563
I20260927 21:28:12.207041 10940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43563 every 8 connection(s)
I20260927 21:28:12.208277 10813 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:28:12.211879 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 10813
I20260927 21:28:12.212114 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-0/wal/instance
I20260927 21:28:12.212926 388 external_mini_cluster.cc:1428] Setting key 6a1b17efa47f069973e2aff1ef1afb39
Sep 27 21:28:12 dist-test-slave-bxm0 krb5kdc[10707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:28:12.228641 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:12.210915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45187 (local address 127.0.97.62:42553)
0927 21:28:12.211393 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:12.211398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:12.212524 (+ 1126us) server_negotiation.cc:407] Connection header received
0927 21:28:12.214432 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:12.214437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:12.214519 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:12.214641 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:12.216182 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:12.216892 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:12.218616 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:12.219022 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:12.219165 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:12.222780 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:28:12.222785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:28:12.222788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:28:12.222821 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:28:12.225396 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:28:12.226299 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:28:12.226304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:28:12.226306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:28:12.226425 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:28:12.226878 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:28:12.226883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:28:12.226890 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:28:12.227152 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:28:12.227306 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:12.228037 (+ 731us) server_negotiation.cc:299] Negotiation successful
0927 21:28:12.228265 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":261,"threads_started":1}
I20260927 21:28:12.229768 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:12.211447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42553 (local address 127.0.97.1:45187)
0927 21:28:12.212360 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0927 21:28:12.212409 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 21:28:12.214156 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:28:12.214881 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:28:12.214890 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:28:12.215406 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:28:12.216001 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:12.216019 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:12.217052 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:12.217057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:12.218455 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:12.218468 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:12.219279 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:12.219286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:12.219525 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:12.220257 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:28:12.220285 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:28:12.222528 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:28:12.225574 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:28:12.225584 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:28:12.225604 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:28:12.226144 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:28:12.226540 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:28:12.226545 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:28:12.226549 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:28:12.226758 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:28:12.227279 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:28:12.227288 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:28:12.227655 (+ 367us) client_negotiation.cc:769] Sending connection context
0927 21:28:12.227929 (+ 274us) client_negotiation.cc:240] Negotiation successful
0927 21:28:12.228506 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":331,"threads_started":1}
I20260927 21:28:12.231557 10941 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42553
I20260927 21:28:12.231945 10941 heartbeater.cc:464] Registering TS with master...
I20260927 21:28:12.233029 10941 heartbeater.cc:511] Master 127.0.97.62:42553 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 21:28:12.235567 10752 ts_manager.cc:194] Registered new tserver with Master: 2e0fb4e8f94141488fddfb40d3e11f4d (127.0.97.1:43563)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 21:28:12.237474 10752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 21:28:12.264985 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42553
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:28:12.595567 10947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:28:12.595897 10947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:28:12.596071 10947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:28:12.603474 10947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:28:12.603637 10947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:28:12.603817 10947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:28:12.613833 10947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:28:12.615164 10947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:28:12.616479 10947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:28:12.630709 10952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:28:12.630656 10955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:28:12.630622 10953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:28:12.631706 10947 server_base.cc:1034] running on GCE node
I20260927 21:28:12.632392 10947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:28:12.633387 10947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:28:12.634626 10947 hybrid_clock.cc:648] HybridClock initialized: now 1790544492634543 us; error 75 us; skew 500 ppm
I20260927 21:28:12.634985 10947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:28:12 dist-test-slave-bxm0 krb5kdc[10707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:28:12.638193 10947 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 21:28:12.639708 10947 webserver.cc:460] Webserver started at http://127.0.97.2:43571/ using document root <none> and password file <none>
I20260927 21:28:12.640398 10947 fs_manager.cc:359] Metadata directory not provided
I20260927 21:28:12.640501 10947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:28:12.640765 10947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:28:12.643772 10947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-1/data/instance:
uuid: "7c759356994a4645a9b40e69a63f6e8e"
format_stamp: "Formatted at 2026-09-27 21:28:12 on dist-test-slave-bxm0"
server_key: "800ebc5642f7f8404aa3a4c6875f6d48"
server_key_iv: "3bce2c6c4b451bf4f89176d1c78a7e6b"
server_key_version: "encryptionkey@0"
I20260927 21:28:12.644506 10947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7c759356994a4645a9b40e69a63f6e8e"
format_stamp: "Formatted at 2026-09-27 21:28:12 on dist-test-slave-bxm0"
server_key: "800ebc5642f7f8404aa3a4c6875f6d48"
server_key_iv: "3bce2c6c4b451bf4f89176d1c78a7e6b"
server_key_version: "encryptionkey@0"
I20260927 21:28:12.649794 10947 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 21:28:12.653554 10962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:12.655054 10947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:28:12.655236 10947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "7c759356994a4645a9b40e69a63f6e8e"
format_stamp: "Formatted at 2026-09-27 21:28:12 on dist-test-slave-bxm0"
server_key: "800ebc5642f7f8404aa3a4c6875f6d48"
server_key_iv: "3bce2c6c4b451bf4f89176d1c78a7e6b"
server_key_version: "encryptionkey@0"
I20260927 21:28:12.655455 10947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:28:12.684278 10947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:28:12.688747 10947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:28:12.689051 10947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:28:12.690042 10947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:28:12.691445 10947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:28:12.691519 10947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:12.691643 10947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:28:12.691692 10947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:12.725513 10947 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38685
I20260927 21:28:12.725574 11074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38685 every 8 connection(s)
I20260927 21:28:12.726872 10947 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:28:12.728572 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 10947
I20260927 21:28:12.728825 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-1/wal/instance
I20260927 21:28:12.729326 388 external_mini_cluster.cc:1428] Setting key aa24967c68ddd26a60898eecad754762
Sep 27 21:28:12 dist-test-slave-bxm0 krb5kdc[10707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:28:12.750149 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:12.729898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33117 (local address 127.0.97.62:42553)
0927 21:28:12.730368 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:12.730374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:12.731681 (+ 1307us) server_negotiation.cc:407] Connection header received
0927 21:28:12.733590 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:12.733598 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:12.733722 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:12.733836 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:12.735488 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:12.736448 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:12.738142 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:12.738860 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:12.739009 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:12.742554 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:28:12.742563 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:28:12.742570 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:28:12.742617 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:28:12.746264 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:28:12.747115 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:28:12.747123 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:28:12.747129 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:28:12.747302 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:28:12.747769 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:28:12.747777 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:28:12.747783 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:28:12.748194 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:28:12.748342 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:12.749355 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0927 21:28:12.749654 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":245,"threads_started":1}
I20260927 21:28:12.751255 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:12.730551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42553 (local address 127.0.97.2:33117)
0927 21:28:12.731476 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0927 21:28:12.731522 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:28:12.733328 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:28:12.734086 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:28:12.734094 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:28:12.734629 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:28:12.735287 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:12.735307 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:12.736603 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:12.736608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:12.738000 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:12.738012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:12.739077 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:12.739082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:12.739289 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:12.740046 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:28:12.740069 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:28:12.742302 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:28:12.746442 (+ 4140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:28:12.746450 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:28:12.746465 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:28:12.746966 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:28:12.747437 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:28:12.747441 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:28:12.747444 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:28:12.747636 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:28:12.748357 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:28:12.748364 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:28:12.748860 (+ 496us) client_negotiation.cc:769] Sending connection context
0927 21:28:12.749208 (+ 348us) client_negotiation.cc:240] Negotiation successful
0927 21:28:12.749756 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":400,"threads_started":1}
I20260927 21:28:12.752923 11075 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42553
I20260927 21:28:12.753230 11075 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 21:28:12.754088 11075 heartbeater.cc:511] Master 127.0.97.62:42553 requested a full tablet report, sending...
I20260927 21:28:12.755625 10752 ts_manager.cc:194] Registered new tserver with Master: 7c759356994a4645a9b40e69a63f6e8e (127.0.97.2:38685)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 21:28:12.756606 10752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33117
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 21:28:12.785763 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42553
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:28:13.119516 11081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:28:13.119877 11081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:28:13.120046 11081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:28:13.127404 11081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:28:13.127568 11081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:28:13.127753 11081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:28:13.138186 11081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:28:13.139523 11081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:28:13.140946 11081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:28:13.154608 11086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:28:13.155124 11089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:28:13.155170 11087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:28:13.155901 11081 server_base.cc:1034] running on GCE node
I20260927 21:28:13.156524 11081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:28:13.157490 11081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:28:13.158747 11081 hybrid_clock.cc:648] HybridClock initialized: now 1790544493158682 us; error 57 us; skew 500 ppm
I20260927 21:28:13.159118 11081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:28:13 dist-test-slave-bxm0 krb5kdc[10707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:28:13.162199 11081 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 21:28:13.163791 11081 webserver.cc:460] Webserver started at http://127.0.97.3:34241/ using document root <none> and password file <none>
I20260927 21:28:13.164502 11081 fs_manager.cc:359] Metadata directory not provided
I20260927 21:28:13.164647 11081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:28:13.164947 11081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:28:13.167914 11081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-2/data/instance:
uuid: "7c403a72daf744ef8663bf0aa9e29bae"
format_stamp: "Formatted at 2026-09-27 21:28:13 on dist-test-slave-bxm0"
server_key: "a34cc76e14fce774a26b3e3f49900a20"
server_key_iv: "b509d312e4f7eaee8abbe0dfaef6fb9f"
server_key_version: "encryptionkey@0"
I20260927 21:28:13.168712 11081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7c403a72daf744ef8663bf0aa9e29bae"
format_stamp: "Formatted at 2026-09-27 21:28:13 on dist-test-slave-bxm0"
server_key: "a34cc76e14fce774a26b3e3f49900a20"
server_key_iv: "b509d312e4f7eaee8abbe0dfaef6fb9f"
server_key_version: "encryptionkey@0"
I20260927 21:28:13.174005 11081 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 21:28:13.177554 11096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:13.179078 11081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:28:13.179271 11081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "7c403a72daf744ef8663bf0aa9e29bae"
format_stamp: "Formatted at 2026-09-27 21:28:13 on dist-test-slave-bxm0"
server_key: "a34cc76e14fce774a26b3e3f49900a20"
server_key_iv: "b509d312e4f7eaee8abbe0dfaef6fb9f"
server_key_version: "encryptionkey@0"
I20260927 21:28:13.179481 11081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:28:13.197883 11081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:28:13.202194 11081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:28:13.202491 11081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:28:13.203449 11081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:28:13.204874 11081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:28:13.204972 11081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:13.205118 11081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:28:13.205188 11081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:13.240490 10941 heartbeater.cc:503] Master 127.0.97.62:42553 was elected leader, sending a full tablet report...
I20260927 21:28:13.241184 11081 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45995
I20260927 21:28:13.241281 11208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45995 every 8 connection(s)
I20260927 21:28:13.242695 11081 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:28:13.248106 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 11081
I20260927 21:28:13.248309 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-2/wal/instance
I20260927 21:28:13.248725 388 external_mini_cluster.cc:1428] Setting key 8966ed443ed6cd5e8841141563ba200a
Sep 27 21:28:13 dist-test-slave-bxm0 krb5kdc[10707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:28:13.263132 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:13.245199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51941 (local address 127.0.97.62:42553)
0927 21:28:13.245355 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:13.245360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:13.246624 (+ 1264us) server_negotiation.cc:407] Connection header received
0927 21:28:13.248408 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:13.248414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:13.248511 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:13.248615 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:13.250750 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.251902 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.253537 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.254002 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.254127 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:13.257513 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:28:13.257518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:28:13.257521 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:28:13.257553 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:28:13.260135 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:28:13.260853 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:28:13.260858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:28:13.260860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:28:13.260984 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:28:13.261361 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:28:13.261364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:28:13.261366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:28:13.261725 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:28:13.261835 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:13.262550 (+ 715us) server_negotiation.cc:299] Negotiation successful
0927 21:28:13.262785 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 21:28:13.264514 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:13.245731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42553 (local address 127.0.97.3:51941)
0927 21:28:13.246458 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0927 21:28:13.246506 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:28:13.248097 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:28:13.248880 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:28:13.248895 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:28:13.249549 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:28:13.250475 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:13.250506 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.252065 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.252070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:13.253386 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:13.253398 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.254243 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.254251 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:13.254474 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:13.255158 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:28:13.255185 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:28:13.257312 (+ 2127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:28:13.260261 (+ 2949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:28:13.260267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:28:13.260282 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:28:13.260717 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:28:13.261082 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:28:13.261085 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:28:13.261087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:28:13.261261 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:28:13.261831 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:28:13.261838 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:28:13.262171 (+ 333us) client_negotiation.cc:769] Sending connection context
0927 21:28:13.262446 (+ 275us) client_negotiation.cc:240] Negotiation successful
0927 21:28:13.262992 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":280,"threads_started":1}
I20260927 21:28:13.265931 11209 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42553
I20260927 21:28:13.266211 11209 heartbeater.cc:464] Registering TS with master...
I20260927 21:28:13.266889 11209 heartbeater.cc:511] Master 127.0.97.62:42553 requested a full tablet report, sending...
I20260927 21:28:13.268324 10752 ts_manager.cc:194] Registered new tserver with Master: 7c403a72daf744ef8663bf0aa9e29bae (127.0.97.3:45995)
I20260927 21:28:13.269152 10752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51941
I20260927 21:28:13.276814 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:28:13.295074 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:13.281267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36652 (local address 127.0.97.62:42553)
0927 21:28:13.281433 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:13.281437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:13.282132 (+ 695us) server_negotiation.cc:407] Connection header received
0927 21:28:13.282369 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:13.282375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:13.282452 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:13.282508 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:13.284599 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.285588 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.286831 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.287229 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.287361 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:13.289673 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:28:13.289678 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:28:13.289682 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:28:13.289717 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:28:13.292359 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:28:13.293025 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:28:13.293031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:28:13.293035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:28:13.293170 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:28:13.293528 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:28:13.293534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:28:13.293537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:28:13.293909 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:28:13.294165 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:13.294489 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 21:28:13.294703 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 21:28:13.299182 10752 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:28:13.306026 10752 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:28:13.336433 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:13.318748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43563 (local address 127.0.0.1:59242)
0927 21:28:13.319791 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0927 21:28:13.319835 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:28:13.319995 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:28:13.321213 (+ 1218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:28:13.321224 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:28:13.321268 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:28:13.321867 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:13.323354 (+ 1487us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.328087 (+ 4733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.328096 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:13.333308 (+ 5212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:13.333341 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.335128 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.335135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:13.335385 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:13.335468 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 21:28:13.335618 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 21:28:13.335981 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"mutex_wait_us":79,"thread_start_us":666,"threads_started":1}
I20260927 21:28:13.337090 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:13.319060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59242 (local address 127.0.97.1:43563)
0927 21:28:13.320454 (+ 1394us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:13.320464 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:13.320493 (+ 29us) server_negotiation.cc:407] Connection header received
0927 21:28:13.320593 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:13.320601 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:13.320890 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:13.323633 (+ 2743us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:28:13.323998 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.327861 (+ 3863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.333561 (+ 5700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.334880 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.335070 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:13.335340 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:13.335673 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 21:28:13.335874 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1243,"thread_start_us":1033,"threads_started":1}
I20260927 21:28:13.340691 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:13.320785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38685 (local address 127.0.0.1:57572)
0927 21:28:13.331246 (+ 10461us) negotiation.cc:107] Waiting for socket to connect
0927 21:28:13.331287 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:28:13.331470 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:28:13.332895 (+ 1425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:28:13.332900 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:28:13.332930 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:28:13.333407 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:13.333422 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.335169 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.335176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:13.337399 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:13.337414 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.339339 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.339346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:13.339529 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:13.339561 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 21:28:13.340163 (+ 602us) client_negotiation.cc:240] Negotiation successful
0927 21:28:13.340311 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10266,"mutex_wait_us":4,"thread_start_us":10812,"threads_started":1}
I20260927 21:28:13.340714 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:13.321257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57572 (local address 127.0.97.2:38685)
0927 21:28:13.332240 (+ 10983us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:13.332250 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:13.332290 (+ 40us) server_negotiation.cc:407] Connection header received
0927 21:28:13.332387 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:13.332393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:13.332697 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:13.332909 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:28:13.333648 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.335005 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.337591 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.339119 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.339247 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:13.339455 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:13.340229 (+ 774us) server_negotiation.cc:299] Negotiation successful
0927 21:28:13.340421 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10822,"thread_start_us":10720,"threads_started":1}
I20260927 21:28:13.341164 10876 tablet_service.cc:1467] Processing CreateTablet for tablet e209a0c3c52940d1927140dc3d933ae0 (DEFAULT_TABLE table=test-table [id=0e0d9451a43648849a3997adb29cf536]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:28:13.342149 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:13.320694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56674 (local address 127.0.97.3:45995)
0927 21:28:13.333764 (+ 13070us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:13.333774 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:13.333807 (+ 33us) server_negotiation.cc:407] Connection header received
0927 21:28:13.333909 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:13.333916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:13.334156 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:13.334370 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:28:13.334991 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.337158 (+ 2167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.339599 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.340831 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.340951 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:13.341182 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:13.341644 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 21:28:13.341798 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12814,"thread_start_us":12665,"threads_started":1}
I20260927 21:28:13.343006 11223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:13.320224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45995 (local address 127.0.0.1:56674)
0927 21:28:13.320730 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 21:28:13.320747 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:28:13.320861 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:28:13.334388 (+ 13527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:28:13.334393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:28:13.334422 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:28:13.334783 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:13.334796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.337423 (+ 2627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.337428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:13.339444 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:13.339459 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.341309 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.341314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:13.341432 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:13.341454 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 21:28:13.341559 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 21:28:13.341728 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":259,"threads_started":1}
I20260927 21:28:13.343082 10876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e209a0c3c52940d1927140dc3d933ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:28:13.344691 11010 tablet_service.cc:1467] Processing CreateTablet for tablet e209a0c3c52940d1927140dc3d933ae0 (DEFAULT_TABLE table=test-table [id=0e0d9451a43648849a3997adb29cf536]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:28:13.346201 11010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e209a0c3c52940d1927140dc3d933ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:28:13.347085 11144 tablet_service.cc:1467] Processing CreateTablet for tablet e209a0c3c52940d1927140dc3d933ae0 (DEFAULT_TABLE table=test-table [id=0e0d9451a43648849a3997adb29cf536]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:28:13.348572 11144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e209a0c3c52940d1927140dc3d933ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:28:13.355213 11227 tablet_bootstrap.cc:492] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d: Bootstrap starting.
I20260927 21:28:13.358383 11227 tablet_bootstrap.cc:654] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d: Neither blocks nor log segments found. Creating new log.
I20260927 21:28:13.359860 11227 log.cc:824] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d: Log is configured to *not* fsync() on all Append() calls
I20260927 21:28:13.362533 11228 tablet_bootstrap.cc:492] T e209a0c3c52940d1927140dc3d933ae0 P 7c759356994a4645a9b40e69a63f6e8e: Bootstrap starting.
I20260927 21:28:13.364265 11230 tablet_bootstrap.cc:492] T e209a0c3c52940d1927140dc3d933ae0 P 7c403a72daf744ef8663bf0aa9e29bae: Bootstrap starting.
I20260927 21:28:13.365581 11227 tablet_bootstrap.cc:492] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d: No bootstrap required, opened a new log
I20260927 21:28:13.365926 11227 ts_tablet_manager.cc:1397] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260927 21:28:13.367224 11230 tablet_bootstrap.cc:654] T e209a0c3c52940d1927140dc3d933ae0 P 7c403a72daf744ef8663bf0aa9e29bae: Neither blocks nor log segments found. Creating new log.
I20260927 21:28:13.367333 11228 tablet_bootstrap.cc:654] T e209a0c3c52940d1927140dc3d933ae0 P 7c759356994a4645a9b40e69a63f6e8e: Neither blocks nor log segments found. Creating new log.
I20260927 21:28:13.368249 11230 log.cc:824] T e209a0c3c52940d1927140dc3d933ae0 P 7c403a72daf744ef8663bf0aa9e29bae: Log is configured to *not* fsync() on all Append() calls
I20260927 21:28:13.368547 11228 log.cc:824] T e209a0c3c52940d1927140dc3d933ae0 P 7c759356994a4645a9b40e69a63f6e8e: Log is configured to *not* fsync() on all Append() calls
I20260927 21:28:13.372414 11228 tablet_bootstrap.cc:492] T e209a0c3c52940d1927140dc3d933ae0 P 7c759356994a4645a9b40e69a63f6e8e: No bootstrap required, opened a new log
I20260927 21:28:13.372691 11228 ts_tablet_manager.cc:1397] T e209a0c3c52940d1927140dc3d933ae0 P 7c759356994a4645a9b40e69a63f6e8e: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 21:28:13.374739 11230 tablet_bootstrap.cc:492] T e209a0c3c52940d1927140dc3d933ae0 P 7c403a72daf744ef8663bf0aa9e29bae: No bootstrap required, opened a new log
I20260927 21:28:13.375008 11230 ts_tablet_manager.cc:1397] T e209a0c3c52940d1927140dc3d933ae0 P 7c403a72daf744ef8663bf0aa9e29bae: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260927 21:28:13.376230 11227 raft_consensus.cc:348] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0fb4e8f94141488fddfb40d3e11f4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43563 } } peers { permanent_uuid: "7c403a72daf744ef8663bf0aa9e29bae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45995 } } peers { permanent_uuid: "7c759356994a4645a9b40e69a63f6e8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38685 } }
I20260927 21:28:13.376693 11227 raft_consensus.cc:374] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:28:13.376791 11227 raft_consensus.cc:729] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e0fb4e8f94141488fddfb40d3e11f4d, State: Initialized, Role: FOLLOWER
I20260927 21:28:13.377424 11227 consensus_queue.cc:260] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0fb4e8f94141488fddfb40d3e11f4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43563 } } peers { permanent_uuid: "7c403a72daf744ef8663bf0aa9e29bae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45995 } } peers { permanent_uuid: "7c759356994a4645a9b40e69a63f6e8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38685 } }
I20260927 21:28:13.379276 11227 ts_tablet_manager.cc:1428] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d: Time spent starting tablet: real 0.013s user 0.007s sys 0.006s
I20260927 21:28:13.382244 11228 raft_consensus.cc:348] T e209a0c3c52940d1927140dc3d933ae0 P 7c759356994a4645a9b40e69a63f6e8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0fb4e8f94141488fddfb40d3e11f4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43563 } } peers { permanent_uuid: "7c403a72daf744ef8663bf0aa9e29bae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45995 } } peers { permanent_uuid: "7c759356994a4645a9b40e69a63f6e8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38685 } }
I20260927 21:28:13.382656 11228 raft_consensus.cc:374] T e209a0c3c52940d1927140dc3d933ae0 P 7c759356994a4645a9b40e69a63f6e8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:28:13.382753 11228 raft_consensus.cc:729] T e209a0c3c52940d1927140dc3d933ae0 P 7c759356994a4645a9b40e69a63f6e8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c759356994a4645a9b40e69a63f6e8e, State: Initialized, Role: FOLLOWER
I20260927 21:28:13.383534 11228 consensus_queue.cc:260] T e209a0c3c52940d1927140dc3d933ae0 P 7c759356994a4645a9b40e69a63f6e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0fb4e8f94141488fddfb40d3e11f4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43563 } } peers { permanent_uuid: "7c403a72daf744ef8663bf0aa9e29bae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45995 } } peers { permanent_uuid: "7c759356994a4645a9b40e69a63f6e8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38685 } }
I20260927 21:28:13.384104 11230 raft_consensus.cc:348] T e209a0c3c52940d1927140dc3d933ae0 P 7c403a72daf744ef8663bf0aa9e29bae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0fb4e8f94141488fddfb40d3e11f4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43563 } } peers { permanent_uuid: "7c403a72daf744ef8663bf0aa9e29bae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45995 } } peers { permanent_uuid: "7c759356994a4645a9b40e69a63f6e8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38685 } }
I20260927 21:28:13.384539 11230 raft_consensus.cc:374] T e209a0c3c52940d1927140dc3d933ae0 P 7c403a72daf744ef8663bf0aa9e29bae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:28:13.384655 11230 raft_consensus.cc:729] T e209a0c3c52940d1927140dc3d933ae0 P 7c403a72daf744ef8663bf0aa9e29bae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c403a72daf744ef8663bf0aa9e29bae, State: Initialized, Role: FOLLOWER
I20260927 21:28:13.385284 11230 consensus_queue.cc:260] T e209a0c3c52940d1927140dc3d933ae0 P 7c403a72daf744ef8663bf0aa9e29bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0fb4e8f94141488fddfb40d3e11f4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43563 } } peers { permanent_uuid: "7c403a72daf744ef8663bf0aa9e29bae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45995 } } peers { permanent_uuid: "7c759356994a4645a9b40e69a63f6e8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38685 } }
I20260927 21:28:13.386003 11075 heartbeater.cc:503] Master 127.0.97.62:42553 was elected leader, sending a full tablet report...
I20260927 21:28:13.386468 11209 heartbeater.cc:503] Master 127.0.97.62:42553 was elected leader, sending a full tablet report...
I20260927 21:28:13.386689 11228 ts_tablet_manager.cc:1428] T e209a0c3c52940d1927140dc3d933ae0 P 7c759356994a4645a9b40e69a63f6e8e: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260927 21:28:13.387089 11230 ts_tablet_manager.cc:1428] T e209a0c3c52940d1927140dc3d933ae0 P 7c403a72daf744ef8663bf0aa9e29bae: Time spent starting tablet: real 0.012s user 0.011s sys 0.002s
W20260927 21:28:13.460641 10942 tablet.cc:2369] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:28:13.479004 11076 tablet.cc:2369] T e209a0c3c52940d1927140dc3d933ae0 P 7c759356994a4645a9b40e69a63f6e8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:28:13.494362 11210 tablet.cc:2369] T e209a0c3c52940d1927140dc3d933ae0 P 7c403a72daf744ef8663bf0aa9e29bae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:28:13.512085 11233 raft_consensus.cc:482] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:28:13.512421 11233 raft_consensus.cc:504] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0fb4e8f94141488fddfb40d3e11f4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43563 } } peers { permanent_uuid: "7c403a72daf744ef8663bf0aa9e29bae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45995 } } peers { permanent_uuid: "7c759356994a4645a9b40e69a63f6e8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38685 } }
I20260927 21:28:13.514238 11233 leader_election.cc:288] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c403a72daf744ef8663bf0aa9e29bae (127.0.97.3:45995), 7c759356994a4645a9b40e69a63f6e8e (127.0.97.2:38685)
I20260927 21:28:13.521519 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:13.514654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34347 (local address 127.0.97.3:45995)
0927 21:28:13.514829 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:13.514834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:13.515386 (+ 552us) server_negotiation.cc:407] Connection header received
0927 21:28:13.515706 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:13.515712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:13.515793 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:13.515898 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:28:13.516600 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.517542 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.519608 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.520489 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.520589 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:13.520664 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:13.521105 (+ 441us) server_negotiation.cc:299] Negotiation successful
0927 21:28:13.521259 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:28:13.521548 11236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:13.514572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45995 (local address 127.0.97.1:34347)
0927 21:28:13.515290 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0927 21:28:13.515312 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:28:13.515500 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:28:13.516006 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:28:13.516011 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:28:13.516040 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:28:13.516454 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:13.516468 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.517704 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.517710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:13.519344 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:13.519355 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.520712 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.520719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:13.520914 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:13.520965 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 21:28:13.521064 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 21:28:13.521202 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":398,"threads_started":1}
I20260927 21:28:13.521864 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:13.515708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37195 (local address 127.0.97.2:38685)
0927 21:28:13.515921 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:13.515927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:13.516193 (+ 266us) server_negotiation.cc:407] Connection header received
0927 21:28:13.516344 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:13.516351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:13.516434 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:13.516525 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:28:13.517120 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.517995 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.519890 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.520849 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.520990 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:13.521099 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:13.521425 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 21:28:13.521578 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 21:28:13.522032 11237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:13.515447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38685 (local address 127.0.97.1:37195)
0927 21:28:13.516096 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0927 21:28:13.516110 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:28:13.516216 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:28:13.516577 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:28:13.516581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:28:13.516604 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:28:13.516964 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:13.516976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.518142 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.518146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:13.519713 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:28:13.519726 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.521083 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.521090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:28:13.521254 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:13.521286 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 21:28:13.521384 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 21:28:13.521500 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":345,"threads_started":1}
I20260927 21:28:13.522774 11164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e209a0c3c52940d1927140dc3d933ae0" candidate_uuid: "2e0fb4e8f94141488fddfb40d3e11f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c403a72daf744ef8663bf0aa9e29bae" is_pre_election: true
I20260927 21:28:13.523085 11030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e209a0c3c52940d1927140dc3d933ae0" candidate_uuid: "2e0fb4e8f94141488fddfb40d3e11f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c759356994a4645a9b40e69a63f6e8e" is_pre_election: true
I20260927 21:28:13.523273 11164 raft_consensus.cc:2454] T e209a0c3c52940d1927140dc3d933ae0 P 7c403a72daf744ef8663bf0aa9e29bae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e0fb4e8f94141488fddfb40d3e11f4d in term 0.
I20260927 21:28:13.523555 11030 raft_consensus.cc:2454] T e209a0c3c52940d1927140dc3d933ae0 P 7c759356994a4645a9b40e69a63f6e8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e0fb4e8f94141488fddfb40d3e11f4d in term 0.
I20260927 21:28:13.524060 10829 leader_election.cc:302] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e0fb4e8f94141488fddfb40d3e11f4d, 7c403a72daf744ef8663bf0aa9e29bae; no voters:
I20260927 21:28:13.524497 11233 raft_consensus.cc:2781] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:28:13.524641 11233 raft_consensus.cc:482] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:28:13.524837 11233 raft_consensus.cc:3037] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:28:13.527072 11233 raft_consensus.cc:504] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0fb4e8f94141488fddfb40d3e11f4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43563 } } peers { permanent_uuid: "7c403a72daf744ef8663bf0aa9e29bae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45995 } } peers { permanent_uuid: "7c759356994a4645a9b40e69a63f6e8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38685 } }
I20260927 21:28:13.527644 11233 leader_election.cc:288] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d [CANDIDATE]: Term 1 election: Requested vote from peers 7c403a72daf744ef8663bf0aa9e29bae (127.0.97.3:45995), 7c759356994a4645a9b40e69a63f6e8e (127.0.97.2:38685)
I20260927 21:28:13.528232 11164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e209a0c3c52940d1927140dc3d933ae0" candidate_uuid: "2e0fb4e8f94141488fddfb40d3e11f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c403a72daf744ef8663bf0aa9e29bae"
I20260927 21:28:13.528322 11030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e209a0c3c52940d1927140dc3d933ae0" candidate_uuid: "2e0fb4e8f94141488fddfb40d3e11f4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c759356994a4645a9b40e69a63f6e8e"
I20260927 21:28:13.528467 11164 raft_consensus.cc:3037] T e209a0c3c52940d1927140dc3d933ae0 P 7c403a72daf744ef8663bf0aa9e29bae [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:28:13.528546 11030 raft_consensus.cc:3037] T e209a0c3c52940d1927140dc3d933ae0 P 7c759356994a4645a9b40e69a63f6e8e [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:28:13.530702 11164 raft_consensus.cc:2454] T e209a0c3c52940d1927140dc3d933ae0 P 7c403a72daf744ef8663bf0aa9e29bae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e0fb4e8f94141488fddfb40d3e11f4d in term 1.
I20260927 21:28:13.530740 11030 raft_consensus.cc:2454] T e209a0c3c52940d1927140dc3d933ae0 P 7c759356994a4645a9b40e69a63f6e8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e0fb4e8f94141488fddfb40d3e11f4d in term 1.
I20260927 21:28:13.531121 10829 leader_election.cc:302] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e0fb4e8f94141488fddfb40d3e11f4d, 7c759356994a4645a9b40e69a63f6e8e; no voters:
I20260927 21:28:13.531454 11233 raft_consensus.cc:2781] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:28:13.531983 11233 raft_consensus.cc:686] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d [term 1 LEADER]: Becoming Leader. State: Replica: 2e0fb4e8f94141488fddfb40d3e11f4d, State: Running, Role: LEADER
I20260927 21:28:13.532523 11233 consensus_queue.cc:237] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0fb4e8f94141488fddfb40d3e11f4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43563 } } peers { permanent_uuid: "7c403a72daf744ef8663bf0aa9e29bae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45995 } } peers { permanent_uuid: "7c759356994a4645a9b40e69a63f6e8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38685 } }
I20260927 21:28:13.536871 10752 catalog_manager.cc:5495] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e0fb4e8f94141488fddfb40d3e11f4d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "2e0fb4e8f94141488fddfb40d3e11f4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e0fb4e8f94141488fddfb40d3e11f4d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c403a72daf744ef8663bf0aa9e29bae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c759356994a4645a9b40e69a63f6e8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38685 } health_report { overall_health: UNKNOWN } } }
I20260927 21:28:13.570327 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:13.564797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59256 (local address 127.0.97.1:43563)
0927 21:28:13.565020 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:13.565029 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:13.565121 (+ 92us) server_negotiation.cc:407] Connection header received
0927 21:28:13.565373 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:13.565377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:13.565498 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:13.565683 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:28:13.566099 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.566802 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.568118 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.568535 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.568667 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:13.569149 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:28:13.569757 (+ 608us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:28:13.569912 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:13.569969 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 21:28:13.570052 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 21:28:13.579254 11164 raft_consensus.cc:1261] T e209a0c3c52940d1927140dc3d933ae0 P 7c403a72daf744ef8663bf0aa9e29bae [term 1 FOLLOWER]: Refusing update from remote peer 2e0fb4e8f94141488fddfb40d3e11f4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:28:13.579254 11030 raft_consensus.cc:1261] T e209a0c3c52940d1927140dc3d933ae0 P 7c759356994a4645a9b40e69a63f6e8e [term 1 FOLLOWER]: Refusing update from remote peer 2e0fb4e8f94141488fddfb40d3e11f4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:28:13.580338 11233 consensus_queue.cc:1035] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c403a72daf744ef8663bf0aa9e29bae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:28:13.580687 11238 consensus_queue.cc:1035] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c759356994a4645a9b40e69a63f6e8e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:28:13.599908 11242 mvcc.cc:205] Tried to move back new op lower bound from 7334070245688664064 to 7334070245512712192. Current Snapshot: MvccSnapshot[applied={T|T < 7334070245688664064}]
I20260927 21:28:13.609323 10752 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:28:13.609649 10752 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:28:13.613816 10752 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5556e1d5c9f4aee9a4b16eb4c62fd02: Sending DeleteTablet for 3 replicas of tablet e209a0c3c52940d1927140dc3d933ae0
I20260927 21:28:13.615437 11010 tablet_service.cc:1514] Processing DeleteTablet for tablet e209a0c3c52940d1927140dc3d933ae0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:28:13 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57572
I20260927 21:28:13.615437 11144 tablet_service.cc:1514] Processing DeleteTablet for tablet e209a0c3c52940d1927140dc3d933ae0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:28:13 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56674
I20260927 21:28:13.615628 10876 tablet_service.cc:1514] Processing DeleteTablet for tablet e209a0c3c52940d1927140dc3d933ae0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:28:13 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59242
I20260927 21:28:13.617184 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 10813
I20260927 21:28:13.628202 11252 tablet_replica.cc:330] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d: stopping tablet replica
I20260927 21:28:13.628732 11252 raft_consensus.cc:2229] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:28:13.629387 11252 raft_consensus.cc:2258] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:28:13.632478 11252 ts_tablet_manager.cc:1905] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:28:13.633169 11251 tablet_replica.cc:330] T e209a0c3c52940d1927140dc3d933ae0 P 7c759356994a4645a9b40e69a63f6e8e: stopping tablet replica
I20260927 21:28:13.633803 11251 raft_consensus.cc:2229] T e209a0c3c52940d1927140dc3d933ae0 P 7c759356994a4645a9b40e69a63f6e8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:28:13.634399 11251 raft_consensus.cc:2258] T e209a0c3c52940d1927140dc3d933ae0 P 7c759356994a4645a9b40e69a63f6e8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:28:13.636816 11251 ts_tablet_manager.cc:1905] T e209a0c3c52940d1927140dc3d933ae0 P 7c759356994a4645a9b40e69a63f6e8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:28:13.640476 11252 ts_tablet_manager.cc:1918] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:28:13.640672 11252 log.cc:1196] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-0/wal/wals/e209a0c3c52940d1927140dc3d933ae0
I20260927 21:28:13.641261 11252 ts_tablet_manager.cc:1939] T e209a0c3c52940d1927140dc3d933ae0 P 2e0fb4e8f94141488fddfb40d3e11f4d: Deleting consensus metadata
I20260927 21:28:13.643235 10736 catalog_manager.cc:4841] TS 2e0fb4e8f94141488fddfb40d3e11f4d (127.0.97.1:43563): tablet e209a0c3c52940d1927140dc3d933ae0 (table test-table [id=0e0d9451a43648849a3997adb29cf536]) successfully deleted
I20260927 21:28:13.644109 11251 ts_tablet_manager.cc:1918] T e209a0c3c52940d1927140dc3d933ae0 P 7c759356994a4645a9b40e69a63f6e8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:28:13.644289 11251 log.cc:1196] T e209a0c3c52940d1927140dc3d933ae0 P 7c759356994a4645a9b40e69a63f6e8e: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-1/wal/wals/e209a0c3c52940d1927140dc3d933ae0
I20260927 21:28:13.644868 11251 ts_tablet_manager.cc:1939] T e209a0c3c52940d1927140dc3d933ae0 P 7c759356994a4645a9b40e69a63f6e8e: Deleting consensus metadata
I20260927 21:28:13.645979 11250 tablet_replica.cc:330] T e209a0c3c52940d1927140dc3d933ae0 P 7c403a72daf744ef8663bf0aa9e29bae: stopping tablet replica
I20260927 21:28:13.646446 11250 raft_consensus.cc:2229] T e209a0c3c52940d1927140dc3d933ae0 P 7c403a72daf744ef8663bf0aa9e29bae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:28:13.646778 10736 catalog_manager.cc:4841] TS 7c759356994a4645a9b40e69a63f6e8e (127.0.97.2:38685): tablet e209a0c3c52940d1927140dc3d933ae0 (table test-table [id=0e0d9451a43648849a3997adb29cf536]) successfully deleted
I20260927 21:28:13.646991 11250 raft_consensus.cc:2258] T e209a0c3c52940d1927140dc3d933ae0 P 7c403a72daf744ef8663bf0aa9e29bae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:28:13.649174 11250 ts_tablet_manager.cc:1905] T e209a0c3c52940d1927140dc3d933ae0 P 7c403a72daf744ef8663bf0aa9e29bae: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 21:28:13 dist-test-slave-bxm0 krb5kdc[10707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 21:28:13.656407 11250 ts_tablet_manager.cc:1918] T e209a0c3c52940d1927140dc3d933ae0 P 7c403a72daf744ef8663bf0aa9e29bae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:28:13.656585 11250 log.cc:1196] T e209a0c3c52940d1927140dc3d933ae0 P 7c403a72daf744ef8663bf0aa9e29bae: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790544390748963-388-0/minicluster-data/ts-2/wal/wals/e209a0c3c52940d1927140dc3d933ae0
I20260927 21:28:13.657143 11250 ts_tablet_manager.cc:1939] T e209a0c3c52940d1927140dc3d933ae0 P 7c403a72daf744ef8663bf0aa9e29bae: Deleting consensus metadata
I20260927 21:28:13.659066 10736 catalog_manager.cc:4841] TS 7c403a72daf744ef8663bf0aa9e29bae (127.0.97.3:45995): tablet e209a0c3c52940d1927140dc3d933ae0 (table test-table [id=0e0d9451a43648849a3997adb29cf536]) successfully deleted
I20260927 21:28:13.661166 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:13.618889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59264 (local address 127.0.97.1:43563)
0927 21:28:13.619124 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:13.619134 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:13.644379 (+ 25245us) server_negotiation.cc:407] Connection header received
0927 21:28:13.644602 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:13.644609 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:13.644735 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:13.644865 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:13.646878 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.647871 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.649240 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.649890 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.650051 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:13.654280 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:28:13.654294 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:28:13.654312 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:28:13.654352 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:28:13.657966 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:28:13.658850 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:28:13.658860 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:28:13.658864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:28:13.659074 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:28:13.659548 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:28:13.659551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:28:13.659553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:28:13.659998 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:28:13.660269 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:13.660612 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 21:28:13.660793 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 21:28:13.661911 10936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:28:13.759829 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 10813
I20260927 21:28:13.777539 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 10947
Sep 27 21:28:13 dist-test-slave-bxm0 krb5kdc[10707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 21:28:13.792436 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:13.778461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57586 (local address 127.0.97.2:38685)
0927 21:28:13.778621 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:13.778626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:13.778644 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:28:13.778738 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:13.778741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:13.778799 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:13.778892 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:13.780010 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.780784 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.782151 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.782561 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.782700 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:13.786884 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:28:13.786901 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:28:13.786917 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:28:13.786964 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:28:13.789810 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:28:13.790476 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:28:13.790483 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:28:13.790485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:28:13.790612 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:28:13.790946 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:28:13.790949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:28:13.790951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:28:13.791339 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:28:13.791575 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:13.791879 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 21:28:13.792071 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:28:13.793119 11070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:28:13.882735 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 10947
I20260927 21:28:13.901995 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 11081
Sep 27 21:28:13 dist-test-slave-bxm0 krb5kdc[10707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 21:28:13.917858 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:13.902820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56676 (local address 127.0.97.3:45995)
0927 21:28:13.903020 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:13.903028 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:13.903050 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:28:13.903274 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:13.903278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:13.903343 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:13.903459 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:13.904717 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.905426 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.906759 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:13.907155 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:13.907291 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:13.911960 (+ 4669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:28:13.911980 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:28:13.911996 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:28:13.912035 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:28:13.915028 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:28:13.915684 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:28:13.915691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:28:13.915694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:28:13.915841 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:28:13.916296 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:28:13.916301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:28:13.916303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:28:13.916739 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:28:13.916979 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:13.917298 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 21:28:13.917459 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 21:28:13.918484 11204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:28:14.006477 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 11081
I20260927 21:28:14.023355 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 10720
I20260927 21:28:14.024508 10782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:28:14.130205 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 10720
2026-09-27T21:28:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4318 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:28:14 dist-test-slave-bxm0 krb5kdc[11260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:28:14 dist-test-slave-bxm0 krb5kdc[11260](info): set up 2 sockets
Sep 27 21:28:14 dist-test-slave-bxm0 krb5kdc[11260](info): commencing operation
krb5kdc: starting...
W20260927 21:28:15.238701 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:28:15 dist-test-slave-bxm0 krb5kdc[11260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:28:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:28:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260927 21:28:15.577404 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasklB5kC7/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/postgres -l logfile start
2026-09-27 21:28:18.067 UTC [11290] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 21:28:18.067 UTC [11290] LOG: listening on IPv4 address "127.0.97.20", port 48461
2026-09-27 21:28:18.071 UTC [11290] LOG: listening on Unix socket "/tmp/.s.PGSQL.48461"
2026-09-27 21:28:18.093 UTC [11293] LOG: database system was shut down at 2026-09-27 21:28:16 UTC
2026-09-27 21:28:18.097 UTC [11290] LOG: database system is ready to accept connections
I20260927 21:28:19.086174 388 mini_postgres.cc:96] Postgres bound to 48461
2026-09-27 21:28:19.104 UTC [11301] FATAL: database "slave" does not exist
127.0.97.20:48461 - accepting connections
I20260927 21:28:19.105151 388 mini_ranger.cc:161] Starting Ranger...
I20260927 21:28:19.133013 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 21:28:19.669248 388 mini_ranger.cc:87] Created ranger Postgres database
I20260927 21:28:19.669387 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ranger-admin
2026-09-27 21:28:19,965 [I] DB FLAVOR :POSTGRES
2026-09-27 21:28:19,965 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 21:28:19,965 [I] Checking connection..
2026-09-27 21:28:19,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 21:28:20,214 [I] Checking connection passed.
2026-09-27 21:28:20,214 [I] --------- Verifying version history table ---------
2026-09-27 21:28:20,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:28:20,483 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 21:28:20,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:28:20,750 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 21:28:20,751 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 21:28:20,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48461/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 21:28:21.000 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 21:28:21,007 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 21:28:21,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:28:21,275 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 21:28:21,276 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 21:28:21,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 21:28:21,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 21:28:21,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-bxm0.c.gcp-upstream.internal','N') ;"
2026-09-27 21:28:22,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 21:28:22,290 [I] Table x_portal_user does not exist in database ranger
2026-09-27 21:28:22,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 21:28:22,542 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 21:28:22,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 21:28:22,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 21:28:23,058 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 21:28:23,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48461/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 21:28:24.058 UTC [11473] WARNING: there is no transaction in progress
2026-09-27 21:28:24.076 UTC [11473] WARNING: there is no transaction in progress
2026-09-27 21:28:24.093 UTC [11473] WARNING: there is no transaction in progress
2026-09-27 21:28:24.109 UTC [11473] WARNING: there is no transaction in progress
2026-09-27 21:28:24.125 UTC [11473] WARNING: there is no transaction in progress
2026-09-27 21:28:24.145 UTC [11473] WARNING: there is no transaction in progress
2026-09-27 21:28:24.213 UTC [11473] WARNING: there is no transaction in progress
2026-09-27 21:28:24.220 UTC [11473] WARNING: there is no transaction in progress
2026-09-27 21:28:24.235 UTC [11473] WARNING: there is no transaction in progress
2026-09-27 21:28:24.247 UTC [11473] WARNING: there is no transaction in progress
2026-09-27 21:28:24.258 UTC [11473] WARNING: there is no transaction in progress
2026-09-27 21:28:24.274 UTC [11473] WARNING: there is no transaction in progress
2026-09-27 21:28:24.285 UTC [11473] WARNING: there is no transaction in progress
2026-09-27 21:28:24.293 UTC [11473] WARNING: there is no transaction in progress
2026-09-27 21:28:24.301 UTC [11473] WARNING: there is no transaction in progress
2026-09-27 21:28:24.308 UTC [11473] WARNING: there is no transaction in progress
2026-09-27 21:28:24,960 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 21:28:24,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 21:28:25,187 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 21:28:25,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 21:28:25,454 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 21:28:25,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 21:28:25,721 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 21:28:25,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 21:28:25,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 21:28:26,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-bxm0.c.gcp-upstream.internal';"
2026-09-27 21:28:26,486 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 21:28:26,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48461/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 21:28:26,750 [I] DB_PATCHES have already been applied
I20260927 21:28:26.756409 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ranger-admin:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasklB5kC7/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 21:28:26.756515 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260927 21:28:26.771633 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:44399
Sep 27, 2026 9:28:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 9:28:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 9:28:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 9:28:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 9:28:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 9:28:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 9:28:27 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 21:28:27.920696 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.816s user 0.000s sys 0.040s
Sep 27, 2026 9:28:48 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 9:28:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 9:28:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 9:28:48 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 9:28:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 9:28:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 9:28:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 9:28:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 9:28:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 9:28:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 9:28:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 9:28:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 9:28:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 9:28:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 9:28:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 9:28:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 9:28:48 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 9:28:49 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 21:28:50.326439 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 21:28:50 dist-test-slave-bxm0 krb5kdc[11260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544530, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 21:28:50.511688 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 21:28:50.533708 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 21:28:51.442090 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 21:28:51.442274 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ranger-kms
I20260927 21:28:51.442425 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:48461
2026-09-27 21:28:51,882 [I] DB FLAVOR :POSTGRES
2026-09-27 21:28:51,883 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 21:28:51,883 [I] Checking connection
2026-09-27 21:28:51,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48461/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 21:28:52,143 [I] connection success
2026-09-27 21:28:52,143 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 21:28:52,143 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 21:28:52,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48461/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 21:28:52,391 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 21:28:52,392 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 21:28:52,392 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 21:28:52,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48461/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 21:28:52,693 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 21:28:53.208371 388 mini_ranger_kms.cc:310] Created kms service
I20260927 21:28:53.380774 388 mini_ranger_kms.cc:326] Created kudu user
I20260927 21:28:53.427201 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 21:28:53 dist-test-slave-bxm0 krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544530, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 21:28:53.866251 388 mini_ranger_kms.cc:384] Added ranger policy
I20260927 21:28:53.866623 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ranger-kms:/tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 21:28:53.866693 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260927 21:28:53.887009 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11654
I20260927 21:28:53.887239 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:58487
Sep 27, 2026 9:28:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 9:28:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 9:28:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 9:28:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 9:28:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 9:28:55 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 21:28:56 dist-test-slave-bxm0 krb5kdc[11260](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790544536, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:28:56.234422 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.723s user 0.116s sys 0.346s
I20260927 21:28:56.234730 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 21:28:56.234937 388 mini_ranger_kms.cc:401] 127.0.97.20:58487/kms/v1/keys
Sep 27 21:28:56 dist-test-slave-bxm0 krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544536, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 21:28:56 dist-test-slave-bxm0 krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544536, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 21:28:59 dist-test-slave-bxm0 krb5kdc[11260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:28:59.152977 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44417
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44417
--ranger_config_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58487/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:28:59.501690 11704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:28:59.502035 11704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:28:59.502204 11704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:28:59.509701 11704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:28:59.510253 11704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:28:59.510360 11704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:28:59.510437 11704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:28:59.510495 11704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:28:59.520917 11704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:58487/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44417
--ranger_config_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:28:59.522352 11704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:28:59.523638 11704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:28:59.535811 11709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:28:59.535743 11712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:28:59.535773 11710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:28:59.537125 11704 server_base.cc:1034] running on GCE node
I20260927 21:28:59.538040 11704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:28:59.539505 11704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:28:59.540726 11704 hybrid_clock.cc:648] HybridClock initialized: now 1790544539540693 us; error 46 us; skew 500 ppm
I20260927 21:28:59.541064 11704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:28:59 dist-test-slave-bxm0 krb5kdc[11260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:28:59.544185 11704 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:28:59.545681 11704 webserver.cc:460] Webserver started at http://127.0.97.62:39625/ using document root <none> and password file <none>
I20260927 21:28:59.546339 11704 fs_manager.cc:359] Metadata directory not provided
I20260927 21:28:59.546478 11704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:28:59.546785 11704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:28:59 dist-test-slave-bxm0 krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 21:28:59.597131 11704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "82557e490bee40778605684c3f4af8cd"
format_stamp: "Formatted at 2026-09-27 21:28:59 on dist-test-slave-bxm0"
server_key: "521bdbf22696ae44891ab6e60574b4fd"
server_key_iv: "8b61ac93182afe9a169c809b1800e23d"
server_key_version: "kuduclusterkey@0"
I20260927 21:28:59.598242 11704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "82557e490bee40778605684c3f4af8cd"
format_stamp: "Formatted at 2026-09-27 21:28:59 on dist-test-slave-bxm0"
server_key: "521bdbf22696ae44891ab6e60574b4fd"
server_key_iv: "8b61ac93182afe9a169c809b1800e23d"
server_key_version: "kuduclusterkey@0"
I20260927 21:28:59.605444 11704 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s
I20260927 21:28:59.627077 11719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:28:59.629942 11704 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260927 21:28:59.630173 11704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "82557e490bee40778605684c3f4af8cd"
format_stamp: "Formatted at 2026-09-27 21:28:59 on dist-test-slave-bxm0"
server_key: "521bdbf22696ae44891ab6e60574b4fd"
server_key_iv: "8b61ac93182afe9a169c809b1800e23d"
server_key_version: "kuduclusterkey@0"
I20260927 21:28:59.630466 11704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:28:59.657191 11704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:28:59.661581 11704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:28:59.661921 11704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:28:59.681430 11704 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44417
I20260927 21:28:59.681452 11770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44417 every 8 connection(s)
I20260927 21:28:59.682881 11704 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:28:59.687134 11771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:28:59.690789 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 11704
I20260927 21:28:59.690995 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:28:59.691636 388 external_mini_cluster.cc:1428] Setting key 7831f1d80cbc846ea3309ccc2f5e9ed7
I20260927 21:28:59.700803 11771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82557e490bee40778605684c3f4af8cd: Bootstrap starting.
I20260927 21:28:59.705005 11771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82557e490bee40778605684c3f4af8cd: Neither blocks nor log segments found. Creating new log.
I20260927 21:28:59.706095 11771 log.cc:824] T 00000000000000000000000000000000 P 82557e490bee40778605684c3f4af8cd: Log is configured to *not* fsync() on all Append() calls
Sep 27 21:28:59 dist-test-slave-bxm0 krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:28:59.709815 11771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82557e490bee40778605684c3f4af8cd: No bootstrap required, opened a new log
I20260927 21:28:59.717101 11774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:28:59.693798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57126 (local address 127.0.97.62:44417)
0927 21:28:59.694627 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0927 21:28:59.694643 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 21:28:59.694693 (+ 50us) server_negotiation.cc:407] Connection header received
0927 21:28:59.696396 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:28:59.696432 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:28:59.696967 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:28:59.697449 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:28:59.698629 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:59.700539 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:59.701906 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:28:59.703467 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:28:59.703648 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:28:59.707833 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:28:59.707845 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:28:59.707866 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:28:59.707906 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:28:59.712124 (+ 4218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:28:59.712931 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:28:59.712941 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:28:59.712951 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:28:59.713183 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:28:59.713582 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:28:59.713587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:28:59.713590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:28:59.714373 (+ 783us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:28:59.714705 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0927 21:28:59.715085 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 21:28:59.715574 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":369,"threads_started":1}
I20260927 21:28:59.721103 11771 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82557e490bee40778605684c3f4af8cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82557e490bee40778605684c3f4af8cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44417 } }
I20260927 21:28:59.721448 11771 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82557e490bee40778605684c3f4af8cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:28:59.721531 11771 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82557e490bee40778605684c3f4af8cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82557e490bee40778605684c3f4af8cd, State: Initialized, Role: FOLLOWER
I20260927 21:28:59.722157 11771 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82557e490bee40778605684c3f4af8cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82557e490bee40778605684c3f4af8cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44417 } }
I20260927 21:28:59.722366 11771 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82557e490bee40778605684c3f4af8cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:28:59.722461 11771 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82557e490bee40778605684c3f4af8cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:28:59.722594 11771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82557e490bee40778605684c3f4af8cd [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:28:59.724665 11771 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82557e490bee40778605684c3f4af8cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82557e490bee40778605684c3f4af8cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44417 } }
I20260927 21:28:59.725111 11771 leader_election.cc:302] T 00000000000000000000000000000000 P 82557e490bee40778605684c3f4af8cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82557e490bee40778605684c3f4af8cd; no voters:
I20260927 21:28:59.725734 11771 leader_election.cc:288] T 00000000000000000000000000000000 P 82557e490bee40778605684c3f4af8cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:28:59.725821 11776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82557e490bee40778605684c3f4af8cd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:28:59.726996 11776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82557e490bee40778605684c3f4af8cd [term 1 LEADER]: Becoming Leader. State: Replica: 82557e490bee40778605684c3f4af8cd, State: Running, Role: LEADER
I20260927 21:28:59.727640 11776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82557e490bee40778605684c3f4af8cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82557e490bee40778605684c3f4af8cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44417 } }
I20260927 21:28:59.728129 11771 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82557e490bee40778605684c3f4af8cd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:28:59.731194 11777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82557e490bee40778605684c3f4af8cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82557e490bee40778605684c3f4af8cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82557e490bee40778605684c3f4af8cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44417 } } }
I20260927 21:28:59.731596 11777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82557e490bee40778605684c3f4af8cd [sys.catalog]: This master's current role is: LEADER
I20260927 21:28:59.731494 11778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82557e490bee40778605684c3f4af8cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82557e490bee40778605684c3f4af8cd. Latest consensus state: current_term: 1 leader_uuid: "82557e490bee40778605684c3f4af8cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82557e490bee40778605684c3f4af8cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44417 } } }
I20260927 21:28:59.731885 11778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82557e490bee40778605684c3f4af8cd [sys.catalog]: This master's current role is: LEADER
I20260927 21:28:59.738433 11782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:28:59.744844 11782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:28:59.745636 11771 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 21:28:59.764029 11782 catalog_manager.cc:1347] Generated new cluster ID: 117c168dcd9443d5be97198cd17e67b2
I20260927 21:28:59.764183 11782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:28:59.783044 11782 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:28:59.785125 11782 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:28:59.798899 11782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82557e490bee40778605684c3f4af8cd: Generated new TSK 0
I20260927 21:28:59.800091 11782 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 21:29:00.550287 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58487/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44417
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:29:00 dist-test-slave-bxm0 krb5kdc[11260](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790544540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 21:29:01.022383 11816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:29:01.022856 11816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:29:01.023039 11816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:29:01.035311 11816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:29:01.035482 11816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:29:01.035755 11816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:29:01.052250 11816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:58487/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:29:01.054085 11816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:29:01.056245 11816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:29:01.078006 11826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:29:01.079725 11827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:29:01.088068 11829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:29:01.089246 11816 server_base.cc:1034] running on GCE node
I20260927 21:29:01.090263 11816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:29:01.091806 11816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:29:01.093119 11816 hybrid_clock.cc:648] HybridClock initialized: now 1790544541093065 us; error 67 us; skew 500 ppm
I20260927 21:29:01.093542 11816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:29:01 dist-test-slave-bxm0 krb5kdc[11260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:29:01.097267 11816 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 21:29:01.101565 11816 webserver.cc:460] Webserver started at http://127.0.97.1:40479/ using document root <none> and password file <none>
I20260927 21:29:01.102653 11816 fs_manager.cc:359] Metadata directory not provided
I20260927 21:29:01.102826 11816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:29:01.103220 11816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:29:01 dist-test-slave-bxm0 krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 21:29:01.129598 11816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-0/data/instance:
uuid: "d76669faa6a348a39ce75b96675f9382"
format_stamp: "Formatted at 2026-09-27 21:29:01 on dist-test-slave-bxm0"
server_key: "3851cd872f332ce59d26c4e942a7e4c1"
server_key_iv: "1e49a341ceac248354f4ecf26ac6393c"
server_key_version: "kuduclusterkey@0"
I20260927 21:29:01.131429 11816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d76669faa6a348a39ce75b96675f9382"
format_stamp: "Formatted at 2026-09-27 21:29:01 on dist-test-slave-bxm0"
server_key: "3851cd872f332ce59d26c4e942a7e4c1"
server_key_iv: "1e49a341ceac248354f4ecf26ac6393c"
server_key_version: "kuduclusterkey@0"
I20260927 21:29:01.144968 11816 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.008s sys 0.004s
I20260927 21:29:01.185761 11836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:29:01.187879 11816 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.011s sys 0.009s
I20260927 21:29:01.188169 11816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "d76669faa6a348a39ce75b96675f9382"
format_stamp: "Formatted at 2026-09-27 21:29:01 on dist-test-slave-bxm0"
server_key: "3851cd872f332ce59d26c4e942a7e4c1"
server_key_iv: "1e49a341ceac248354f4ecf26ac6393c"
server_key_version: "kuduclusterkey@0"
I20260927 21:29:01.188546 11816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:29:01.219471 11816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:29:01.224452 11816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:29:01.224803 11816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:29:01.238842 11816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:29:01.240975 11816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:29:01.241114 11816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:29:01.241307 11816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:29:01.241376 11816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:29:01 dist-test-slave-bxm0 krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 21:29:01.767741 11816 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40555
I20260927 21:29:01.767813 11948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40555 every 8 connection(s)
I20260927 21:29:01.769556 11816 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:29:01.779299 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 11816
I20260927 21:29:01.779587 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-0/wal/instance
I20260927 21:29:01.780610 388 external_mini_cluster.cc:1428] Setting key 127be7ad051906cfb70ceec3688dceeb
Sep 27 21:29:01 dist-test-slave-bxm0 krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:29:01.809376 11951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:01.772785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37873 (local address 127.0.97.62:44417)
0927 21:29:01.785405 (+ 12620us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:01.785412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:01.789990 (+ 4578us) server_negotiation.cc:407] Connection header received
0927 21:29:01.790082 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:01.790088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:01.790174 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:01.790283 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:29:01.792634 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:01.793868 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:01.796499 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:01.797076 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:01.797259 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:01.802178 (+ 4919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:29:01.802185 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:29:01.802189 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:29:01.802230 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:29:01.805831 (+ 3601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:01.806770 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:01.806775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:01.806779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:01.806922 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:01.807420 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:01.807424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:01.807427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:01.807749 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:29:01.807855 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:01.808801 (+ 946us) server_negotiation.cc:299] Negotiation successful
0927 21:29:01.809022 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12486,"thread_start_us":12332,"threads_started":1}
I20260927 21:29:01.811793 11952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:01.773427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44417 (local address 127.0.97.1:37873)
0927 21:29:01.786073 (+ 12646us) negotiation.cc:107] Waiting for socket to connect
0927 21:29:01.786136 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 21:29:01.788627 (+ 2491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:29:01.790758 (+ 2131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:29:01.790771 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:29:01.791456 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:29:01.792405 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:01.792431 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:01.794112 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:01.794117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:01.796244 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:01.796271 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:01.797680 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:01.797689 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:01.797962 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:01.798753 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:29:01.798782 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:29:01.801877 (+ 3095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:29:01.806032 (+ 4155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:29:01.806040 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:29:01.806070 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:29:01.806629 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:29:01.807052 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:29:01.807067 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:29:01.807070 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:29:01.807301 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:29:01.807950 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:29:01.807958 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:29:01.808398 (+ 440us) client_negotiation.cc:769] Sending connection context
0927 21:29:01.809926 (+ 1528us) client_negotiation.cc:240] Negotiation successful
0927 21:29:01.810462 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12258,"thread_start_us":15603,"threads_started":1}
I20260927 21:29:01.814196 11949 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44417
I20260927 21:29:01.814499 11949 heartbeater.cc:464] Registering TS with master...
I20260927 21:29:01.815307 11949 heartbeater.cc:511] Master 127.0.97.62:44417 requested a full tablet report, sending...
I20260927 21:29:01.817703 11736 ts_manager.cc:194] Registered new tserver with Master: d76669faa6a348a39ce75b96675f9382 (127.0.97.1:40555)
I20260927 21:29:01.819552 11736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37873
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 21:29:01.873540 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58487/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44417
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:29:02 dist-test-slave-bxm0 krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260927 21:29:02.337306 11955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:29:02.337595 11955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:29:02.337774 11955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:29:02.345710 11955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:29:02.345824 11955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:29:02.346010 11955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:29:02.355959 11955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:58487/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:29:02.357255 11955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:29:02.358636 11955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:29:02.372390 11969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:29:02.372457 11972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:29:02.372435 11970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:29:02.374956 11955 server_base.cc:1034] running on GCE node
I20260927 21:29:02.375617 11955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:29:02.376684 11955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:29:02.377933 11955 hybrid_clock.cc:648] HybridClock initialized: now 1790544542377889 us; error 64 us; skew 500 ppm
I20260927 21:29:02.378263 11955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:29:02 dist-test-slave-bxm0 krb5kdc[11260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:29:02.381268 11955 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 21:29:02.382889 11955 webserver.cc:460] Webserver started at http://127.0.97.2:37045/ using document root <none> and password file <none>
I20260927 21:29:02.383714 11955 fs_manager.cc:359] Metadata directory not provided
I20260927 21:29:02.383867 11955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:29:02.384186 11955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:29:02 dist-test-slave-bxm0 krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 21:29:02.398638 11955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-1/data/instance:
uuid: "f6c6104cd9c24ce7a0e04f247fc13ccb"
format_stamp: "Formatted at 2026-09-27 21:29:02 on dist-test-slave-bxm0"
server_key: "a61a0ab5df62473664fdef1fe7c7554a"
server_key_iv: "5036692d54909cd0b1f555347d4a6023"
server_key_version: "kuduclusterkey@0"
I20260927 21:29:02.399495 11955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f6c6104cd9c24ce7a0e04f247fc13ccb"
format_stamp: "Formatted at 2026-09-27 21:29:02 on dist-test-slave-bxm0"
server_key: "a61a0ab5df62473664fdef1fe7c7554a"
server_key_iv: "5036692d54909cd0b1f555347d4a6023"
server_key_version: "kuduclusterkey@0"
I20260927 21:29:02.404662 11955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 21:29:02.419031 11979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:29:02.420665 11955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:29:02.420866 11955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "f6c6104cd9c24ce7a0e04f247fc13ccb"
format_stamp: "Formatted at 2026-09-27 21:29:02 on dist-test-slave-bxm0"
server_key: "a61a0ab5df62473664fdef1fe7c7554a"
server_key_iv: "5036692d54909cd0b1f555347d4a6023"
server_key_version: "kuduclusterkey@0"
I20260927 21:29:02.421094 11955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:29:02.449528 11955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:29:02.453716 11955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:29:02.454013 11955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:29:02.454974 11955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:29:02.456311 11955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:29:02.456413 11955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:29:02.456561 11955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:29:02.456641 11955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:29:02.493798 11955 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36219
I20260927 21:29:02.493896 12091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36219 every 8 connection(s)
I20260927 21:29:02.495110 11955 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:29:02.504328 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 11955
I20260927 21:29:02.504518 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-1/wal/instance
I20260927 21:29:02.504945 388 external_mini_cluster.cc:1428] Setting key 8c30209ff5486d1c4ed7c535cded7f60
Sep 27 21:29:02 dist-test-slave-bxm0 krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:29:02.516171 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:02.498338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40383 (local address 127.0.97.62:44417)
0927 21:29:02.498816 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:02.498820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:02.499630 (+ 810us) server_negotiation.cc:407] Connection header received
0927 21:29:02.501435 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:02.501441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:02.501540 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:02.501681 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:29:02.503176 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:02.503936 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:02.505714 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:02.506128 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:02.506262 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:02.509878 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:29:02.509884 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:29:02.509887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:29:02.509921 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:29:02.512536 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:02.513315 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:02.513322 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:02.513328 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:02.513454 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:02.513933 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:02.513940 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:02.513943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:02.514218 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:29:02.514376 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:02.515131 (+ 755us) server_negotiation.cc:299] Negotiation successful
0927 21:29:02.515362 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":271,"threads_started":1}
I20260927 21:29:02.517227 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:02.498742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44417 (local address 127.0.97.2:40383)
0927 21:29:02.499450 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0927 21:29:02.499503 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 21:29:02.501154 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:29:02.501945 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:29:02.501953 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:29:02.502386 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:29:02.502948 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:02.502977 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:02.504104 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:02.504114 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:02.505531 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:02.505544 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:02.506369 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:02.506377 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:02.506622 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:02.507319 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:29:02.507345 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:29:02.509569 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:29:02.512719 (+ 3150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:29:02.512729 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:29:02.512748 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:29:02.513191 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:29:02.513567 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:29:02.513573 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:29:02.513579 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:29:02.513818 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:29:02.514351 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:29:02.514360 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:29:02.514707 (+ 347us) client_negotiation.cc:769] Sending connection context
0927 21:29:02.515025 (+ 318us) client_negotiation.cc:240] Negotiation successful
0927 21:29:02.515963 (+ 938us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":289,"threads_started":1}
I20260927 21:29:02.518796 12092 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44417
I20260927 21:29:02.519114 12092 heartbeater.cc:464] Registering TS with master...
I20260927 21:29:02.519829 12092 heartbeater.cc:511] Master 127.0.97.62:44417 requested a full tablet report, sending...
I20260927 21:29:02.521410 11736 ts_manager.cc:194] Registered new tserver with Master: f6c6104cd9c24ce7a0e04f247fc13ccb (127.0.97.2:36219)
I20260927 21:29:02.522716 11736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40383
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 21:29:02.556764 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:58487/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44417
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:29:02.823856 11949 heartbeater.cc:503] Master 127.0.97.62:44417 was elected leader, sending a full tablet report...
W20260927 21:29:02.917268 12098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:29:02.917572 12098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:29:02.917765 12098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:29:02.926121 12098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:29:02.926239 12098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:29:02.926350 12098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:29:02.936697 12098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:58487/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:29:02.938081 12098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:29:02.939560 12098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:29:02.953635 12103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:29:02.953500 12106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:29:02.953549 12104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:29:02.954344 12098 server_base.cc:1034] running on GCE node
I20260927 21:29:02.955055 12098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:29:02.956089 12098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:29:02.957363 12098 hybrid_clock.cc:648] HybridClock initialized: now 1790544542957304 us; error 53 us; skew 500 ppm
I20260927 21:29:02.957811 12098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:29:02 dist-test-slave-bxm0 krb5kdc[11260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:29:02.961138 12098 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 21:29:02.962718 12098 webserver.cc:460] Webserver started at http://127.0.97.3:39051/ using document root <none> and password file <none>
I20260927 21:29:02.963388 12098 fs_manager.cc:359] Metadata directory not provided
I20260927 21:29:02.963498 12098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:29:02.963770 12098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:29:02 dist-test-slave-bxm0 krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 21:29:02.981323 12098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-2/data/instance:
uuid: "ada1bde3127445c2a898f7d7a85a157a"
format_stamp: "Formatted at 2026-09-27 21:29:02 on dist-test-slave-bxm0"
server_key: "8c1c0024d83ed1e189f684e80d8fc5f2"
server_key_iv: "39d44e8c37d9fb88e37bc51485f9b534"
server_key_version: "kuduclusterkey@0"
I20260927 21:29:02.982537 12098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ada1bde3127445c2a898f7d7a85a157a"
format_stamp: "Formatted at 2026-09-27 21:29:02 on dist-test-slave-bxm0"
server_key: "8c1c0024d83ed1e189f684e80d8fc5f2"
server_key_iv: "39d44e8c37d9fb88e37bc51485f9b534"
server_key_version: "kuduclusterkey@0"
I20260927 21:29:02.989697 12098 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260927 21:29:03.005985 12113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:29:03.007834 12098 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 21:29:03.008033 12098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "ada1bde3127445c2a898f7d7a85a157a"
format_stamp: "Formatted at 2026-09-27 21:29:02 on dist-test-slave-bxm0"
server_key: "8c1c0024d83ed1e189f684e80d8fc5f2"
server_key_iv: "39d44e8c37d9fb88e37bc51485f9b534"
server_key_version: "kuduclusterkey@0"
I20260927 21:29:03.008308 12098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:29:03.038888 12098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:29:03.042958 12098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:29:03.043215 12098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:29:03.044154 12098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:29:03.045547 12098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:29:03.045645 12098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 21:29:03.045732 12098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:29:03.045778 12098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:29:03.082624 12098 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44771
I20260927 21:29:03.082664 12225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44771 every 8 connection(s)
I20260927 21:29:03.084020 12098 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:29:03.091178 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 12098
I20260927 21:29:03.091372 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-2/wal/instance
I20260927 21:29:03.091876 388 external_mini_cluster.cc:1428] Setting key a6362a0ef214fbcba3dcaec227a5efd8
Sep 27 21:29:03 dist-test-slave-bxm0 krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:29:03.105150 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:03.086727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47983 (local address 127.0.97.62:44417)
0927 21:29:03.087230 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:03.087235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:03.088194 (+ 959us) server_negotiation.cc:407] Connection header received
0927 21:29:03.090107 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:03.090113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:03.090209 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:03.090333 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:29:03.092004 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.092934 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.095130 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.095666 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.095769 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:03.099275 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:29:03.099281 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:29:03.099284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:29:03.099328 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:29:03.102349 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:03.103091 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:03.103095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:03.103098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:03.103220 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:03.103636 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:03.103639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:03.103641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:03.103907 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:29:03.104074 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:03.104729 (+ 655us) server_negotiation.cc:299] Negotiation successful
0927 21:29:03.104897 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":257,"threads_started":1}
I20260927 21:29:03.106457 12229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:03.087221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44417 (local address 127.0.97.3:47983)
0927 21:29:03.088008 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0927 21:29:03.088061 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 21:29:03.089802 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:29:03.090588 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:29:03.090597 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:29:03.091063 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:29:03.091814 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:03.091833 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.093119 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.093123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:03.094925 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:03.094968 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.095870 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.095877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:03.096102 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:03.096874 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:29:03.096902 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:29:03.099033 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:29:03.102495 (+ 3462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:29:03.102503 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:29:03.102518 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:29:03.102968 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:29:03.103314 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:29:03.103317 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:29:03.103319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:29:03.103520 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:29:03.104076 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:29:03.104082 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:29:03.104412 (+ 330us) client_negotiation.cc:769] Sending connection context
0927 21:29:03.104641 (+ 229us) client_negotiation.cc:240] Negotiation successful
0927 21:29:03.105024 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":319,"threads_started":1}
I20260927 21:29:03.107867 12226 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44417
I20260927 21:29:03.108182 12226 heartbeater.cc:464] Registering TS with master...
I20260927 21:29:03.108881 12226 heartbeater.cc:511] Master 127.0.97.62:44417 requested a full tablet report, sending...
I20260927 21:29:03.110309 11736 ts_manager.cc:194] Registered new tserver with Master: ada1bde3127445c2a898f7d7a85a157a (127.0.97.3:44771)
I20260927 21:29:03.111099 11736 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47983
I20260927 21:29:03.120003 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:29:03.139350 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:03.126135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51106 (local address 127.0.97.62:44417)
0927 21:29:03.126376 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:03.126386 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:03.126759 (+ 373us) server_negotiation.cc:407] Connection header received
0927 21:29:03.126897 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:03.126901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:03.126977 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:03.127063 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:29:03.128565 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.129292 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.130813 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.131239 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.131333 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:03.133720 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:29:03.133725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:29:03.133728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:29:03.133759 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:29:03.136788 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:03.137458 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:03.137462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:03.137464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:03.137582 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:03.137962 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:03.137965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:03.137967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:03.138217 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:29:03.138383 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:03.138709 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 21:29:03.138951 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 21:29:03.143708 11736 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:29:03.150826 11736 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:29:03.191566 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:03.164026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59440 (local address 127.0.97.1:40555)
0927 21:29:03.164851 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:03.164865 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:03.180423 (+ 15558us) server_negotiation.cc:407] Connection header received
0927 21:29:03.181013 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:03.181026 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:03.181404 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:03.181644 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:29:03.182933 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.185183 (+ 2250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.188847 (+ 3664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.190361 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.190517 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:03.190778 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:03.191037 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 21:29:03.191219 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":844,"threads_started":1}
I20260927 21:29:03.192126 12243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:03.163688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40555 (local address 127.0.0.1:59440)
0927 21:29:03.178945 (+ 15257us) negotiation.cc:107] Waiting for socket to connect
0927 21:29:03.180212 (+ 1267us) client_negotiation.cc:174] Beginning negotiation
0927 21:29:03.180680 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:29:03.182069 (+ 1389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:29:03.182081 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:29:03.182124 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:29:03.182660 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:03.182677 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.185440 (+ 2763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.185447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:03.188660 (+ 3213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:03.188675 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.190585 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.190592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:03.190790 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:03.190848 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 21:29:03.190978 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 21:29:03.191189 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14839,"thread_start_us":16748,"threads_started":1}
I20260927 21:29:03.192966 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:03.163838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44771 (local address 127.0.0.1:49394)
0927 21:29:03.180736 (+ 16898us) negotiation.cc:107] Waiting for socket to connect
0927 21:29:03.180755 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:29:03.180881 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:29:03.185014 (+ 4133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:29:03.185019 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:29:03.185043 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:29:03.185521 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:03.185535 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.187271 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.187278 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:03.190003 (+ 2725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:03.190019 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.192131 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.192137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:03.192331 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:03.192373 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 21:29:03.192456 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 21:29:03.192587 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16758,"mutex_wait_us":6,"thread_start_us":22066,"threads_started":1}
I20260927 21:29:03.193447 12242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:03.164626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49394 (local address 127.0.97.3:44771)
0927 21:29:03.184082 (+ 19456us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:03.184092 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:03.184125 (+ 33us) server_negotiation.cc:407] Connection header received
0927 21:29:03.184220 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:03.184228 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:03.184489 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:03.184726 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:29:03.185795 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.187142 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.190173 (+ 3031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.191625 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.191752 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:03.191986 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:03.192988 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0927 21:29:03.193148 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19258,"thread_start_us":20271,"threads_started":1}
I20260927 21:29:03.194703 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:03.165848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36219 (local address 127.0.0.1:60036)
0927 21:29:03.186230 (+ 20382us) negotiation.cc:107] Waiting for socket to connect
0927 21:29:03.186250 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:29:03.186378 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:29:03.186963 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:29:03.186968 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:29:03.186992 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:29:03.188081 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:03.188098 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.189571 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.189577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:03.191737 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:03.191750 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.193897 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.193905 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:03.194082 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:03.194113 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 21:29:03.194209 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 21:29:03.194335 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20246,"mutex_wait_us":21,"thread_start_us":13276,"threads_started":1}
I20260927 21:29:03.194766 12244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:03.166109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60036 (local address 127.0.97.2:36219)
0927 21:29:03.185527 (+ 19418us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:03.185536 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:03.186379 (+ 843us) server_negotiation.cc:407] Connection header received
0927 21:29:03.186512 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:03.186520 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:03.186756 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:03.186961 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:29:03.188284 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.189418 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.191906 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.193382 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.193508 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:03.193721 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:03.194277 (+ 556us) server_negotiation.cc:299] Negotiation successful
0927 21:29:03.194451 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19230,"thread_start_us":19420,"threads_started":1}
I20260927 21:29:03.196964 11884 tablet_service.cc:1467] Processing CreateTablet for tablet ee7d49b432fb4c9783338e812dbd5354 (DEFAULT_TABLE table=test-table [id=fb7eb2ec82c546dda517310f4881b6ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:29:03.197306 12161 tablet_service.cc:1467] Processing CreateTablet for tablet ee7d49b432fb4c9783338e812dbd5354 (DEFAULT_TABLE table=test-table [id=fb7eb2ec82c546dda517310f4881b6ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:29:03.198571 12161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee7d49b432fb4c9783338e812dbd5354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:29:03.198654 11884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee7d49b432fb4c9783338e812dbd5354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:29:03.198544 12027 tablet_service.cc:1467] Processing CreateTablet for tablet ee7d49b432fb4c9783338e812dbd5354 (DEFAULT_TABLE table=test-table [id=fb7eb2ec82c546dda517310f4881b6ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:29:03.199880 12027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee7d49b432fb4c9783338e812dbd5354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:29:03.210263 12245 tablet_bootstrap.cc:492] T ee7d49b432fb4c9783338e812dbd5354 P ada1bde3127445c2a898f7d7a85a157a: Bootstrap starting.
I20260927 21:29:03.213572 12246 tablet_bootstrap.cc:492] T ee7d49b432fb4c9783338e812dbd5354 P d76669faa6a348a39ce75b96675f9382: Bootstrap starting.
I20260927 21:29:03.215261 12245 tablet_bootstrap.cc:654] T ee7d49b432fb4c9783338e812dbd5354 P ada1bde3127445c2a898f7d7a85a157a: Neither blocks nor log segments found. Creating new log.
I20260927 21:29:03.216605 12245 log.cc:824] T ee7d49b432fb4c9783338e812dbd5354 P ada1bde3127445c2a898f7d7a85a157a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:29:03.217980 12246 tablet_bootstrap.cc:654] T ee7d49b432fb4c9783338e812dbd5354 P d76669faa6a348a39ce75b96675f9382: Neither blocks nor log segments found. Creating new log.
I20260927 21:29:03.219347 12246 log.cc:824] T ee7d49b432fb4c9783338e812dbd5354 P d76669faa6a348a39ce75b96675f9382: Log is configured to *not* fsync() on all Append() calls
I20260927 21:29:03.220122 12245 tablet_bootstrap.cc:492] T ee7d49b432fb4c9783338e812dbd5354 P ada1bde3127445c2a898f7d7a85a157a: No bootstrap required, opened a new log
I20260927 21:29:03.220108 12247 tablet_bootstrap.cc:492] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb: Bootstrap starting.
I20260927 21:29:03.220990 12245 ts_tablet_manager.cc:1397] T ee7d49b432fb4c9783338e812dbd5354 P ada1bde3127445c2a898f7d7a85a157a: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260927 21:29:03.222543 12246 tablet_bootstrap.cc:492] T ee7d49b432fb4c9783338e812dbd5354 P d76669faa6a348a39ce75b96675f9382: No bootstrap required, opened a new log
I20260927 21:29:03.222826 12246 ts_tablet_manager.cc:1397] T ee7d49b432fb4c9783338e812dbd5354 P d76669faa6a348a39ce75b96675f9382: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260927 21:29:03.224156 12247 tablet_bootstrap.cc:654] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb: Neither blocks nor log segments found. Creating new log.
I20260927 21:29:03.225407 12247 log.cc:824] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb: Log is configured to *not* fsync() on all Append() calls
I20260927 21:29:03.231655 12246 raft_consensus.cc:348] T ee7d49b432fb4c9783338e812dbd5354 P d76669faa6a348a39ce75b96675f9382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d76669faa6a348a39ce75b96675f9382" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40555 } } peers { permanent_uuid: "ada1bde3127445c2a898f7d7a85a157a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44771 } } peers { permanent_uuid: "f6c6104cd9c24ce7a0e04f247fc13ccb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } }
I20260927 21:29:03.232074 12246 raft_consensus.cc:374] T ee7d49b432fb4c9783338e812dbd5354 P d76669faa6a348a39ce75b96675f9382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:29:03.232213 12246 raft_consensus.cc:729] T ee7d49b432fb4c9783338e812dbd5354 P d76669faa6a348a39ce75b96675f9382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d76669faa6a348a39ce75b96675f9382, State: Initialized, Role: FOLLOWER
I20260927 21:29:03.232879 12246 consensus_queue.cc:260] T ee7d49b432fb4c9783338e812dbd5354 P d76669faa6a348a39ce75b96675f9382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d76669faa6a348a39ce75b96675f9382" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40555 } } peers { permanent_uuid: "ada1bde3127445c2a898f7d7a85a157a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44771 } } peers { permanent_uuid: "f6c6104cd9c24ce7a0e04f247fc13ccb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } }
I20260927 21:29:03.233408 12247 tablet_bootstrap.cc:492] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb: No bootstrap required, opened a new log
I20260927 21:29:03.233124 12245 raft_consensus.cc:348] T ee7d49b432fb4c9783338e812dbd5354 P ada1bde3127445c2a898f7d7a85a157a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d76669faa6a348a39ce75b96675f9382" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40555 } } peers { permanent_uuid: "ada1bde3127445c2a898f7d7a85a157a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44771 } } peers { permanent_uuid: "f6c6104cd9c24ce7a0e04f247fc13ccb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } }
I20260927 21:29:03.233551 12245 raft_consensus.cc:374] T ee7d49b432fb4c9783338e812dbd5354 P ada1bde3127445c2a898f7d7a85a157a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:29:03.233693 12245 raft_consensus.cc:729] T ee7d49b432fb4c9783338e812dbd5354 P ada1bde3127445c2a898f7d7a85a157a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada1bde3127445c2a898f7d7a85a157a, State: Initialized, Role: FOLLOWER
I20260927 21:29:03.234019 12247 ts_tablet_manager.cc:1397] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.006s
I20260927 21:29:03.234440 12245 consensus_queue.cc:260] T ee7d49b432fb4c9783338e812dbd5354 P ada1bde3127445c2a898f7d7a85a157a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d76669faa6a348a39ce75b96675f9382" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40555 } } peers { permanent_uuid: "ada1bde3127445c2a898f7d7a85a157a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44771 } } peers { permanent_uuid: "f6c6104cd9c24ce7a0e04f247fc13ccb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } }
I20260927 21:29:03.234747 12246 ts_tablet_manager.cc:1428] T ee7d49b432fb4c9783338e812dbd5354 P d76669faa6a348a39ce75b96675f9382: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260927 21:29:03.242339 12226 heartbeater.cc:503] Master 127.0.97.62:44417 was elected leader, sending a full tablet report...
I20260927 21:29:03.244664 12245 ts_tablet_manager.cc:1428] T ee7d49b432fb4c9783338e812dbd5354 P ada1bde3127445c2a898f7d7a85a157a: Time spent starting tablet: real 0.023s user 0.014s sys 0.007s
I20260927 21:29:03.247216 12247 raft_consensus.cc:348] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d76669faa6a348a39ce75b96675f9382" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40555 } } peers { permanent_uuid: "ada1bde3127445c2a898f7d7a85a157a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44771 } } peers { permanent_uuid: "f6c6104cd9c24ce7a0e04f247fc13ccb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } }
I20260927 21:29:03.247768 12247 raft_consensus.cc:374] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:29:03.247886 12247 raft_consensus.cc:729] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c6104cd9c24ce7a0e04f247fc13ccb, State: Initialized, Role: FOLLOWER
I20260927 21:29:03.248461 12247 consensus_queue.cc:260] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d76669faa6a348a39ce75b96675f9382" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40555 } } peers { permanent_uuid: "ada1bde3127445c2a898f7d7a85a157a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44771 } } peers { permanent_uuid: "f6c6104cd9c24ce7a0e04f247fc13ccb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } }
I20260927 21:29:03.249424 12092 heartbeater.cc:503] Master 127.0.97.62:44417 was elected leader, sending a full tablet report...
I20260927 21:29:03.249972 12247 ts_tablet_manager.cc:1428] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
W20260927 21:29:03.274035 11950 tablet.cc:2369] T ee7d49b432fb4c9783338e812dbd5354 P d76669faa6a348a39ce75b96675f9382: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:29:03.283973 12253 raft_consensus.cc:482] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:29:03.284282 12253 raft_consensus.cc:504] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d76669faa6a348a39ce75b96675f9382" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40555 } } peers { permanent_uuid: "ada1bde3127445c2a898f7d7a85a157a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44771 } } peers { permanent_uuid: "f6c6104cd9c24ce7a0e04f247fc13ccb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } }
I20260927 21:29:03.286079 12253 leader_election.cc:288] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d76669faa6a348a39ce75b96675f9382 (127.0.97.1:40555), ada1bde3127445c2a898f7d7a85a157a (127.0.97.3:44771)
I20260927 21:29:03.295408 12242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:03.286647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47279 (local address 127.0.97.3:44771)
0927 21:29:03.286916 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:03.286924 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:03.287473 (+ 549us) server_negotiation.cc:407] Connection header received
0927 21:29:03.287702 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:03.287706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:03.287770 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:03.287871 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:29:03.288632 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.289564 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.292762 (+ 3198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.294115 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.294239 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:03.294352 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:03.295016 (+ 664us) server_negotiation.cc:299] Negotiation successful
0927 21:29:03.295157 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 21:29:03.295415 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:03.286448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54581 (local address 127.0.97.1:40555)
0927 21:29:03.286696 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:03.286705 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:03.287456 (+ 751us) server_negotiation.cc:407] Connection header received
0927 21:29:03.287694 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:03.287698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:03.287761 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:03.287858 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:29:03.288558 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.289464 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.292766 (+ 3302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.294085 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.294240 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:03.294358 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:03.295016 (+ 658us) server_negotiation.cc:299] Negotiation successful
0927 21:29:03.295156 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 21:29:03.295754 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:03.286313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40555 (local address 127.0.97.2:54581)
0927 21:29:03.287316 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0927 21:29:03.287340 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 21:29:03.287538 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:29:03.287883 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:29:03.287887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:29:03.287930 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:29:03.288371 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:03.288388 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.289645 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.289654 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:03.292561 (+ 2907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:03.292579 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.294437 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.294445 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:03.294727 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:03.294866 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 21:29:03.294986 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 21:29:03.295187 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"mutex_wait_us":203,"spinlock_wait_cycles":33152,"thread_start_us":583,"threads_started":1}
I20260927 21:29:03.295778 12255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:03.286458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44771 (local address 127.0.97.2:47279)
0927 21:29:03.287364 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0927 21:29:03.287378 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:29:03.287554 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:29:03.287884 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:29:03.287888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:29:03.287936 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:29:03.288423 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:03.288432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.289714 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.289718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:03.292582 (+ 2864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:03.292590 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.294438 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.294445 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:03.294727 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:03.294865 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 21:29:03.294973 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 21:29:03.295187 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"mutex_wait_us":225,"spinlock_wait_cycles":11648,"thread_start_us":633,"threads_started":1}
I20260927 21:29:03.297132 12181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee7d49b432fb4c9783338e812dbd5354" candidate_uuid: "f6c6104cd9c24ce7a0e04f247fc13ccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada1bde3127445c2a898f7d7a85a157a" is_pre_election: true
I20260927 21:29:03.297125 11904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee7d49b432fb4c9783338e812dbd5354" candidate_uuid: "f6c6104cd9c24ce7a0e04f247fc13ccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d76669faa6a348a39ce75b96675f9382" is_pre_election: true
I20260927 21:29:03.297750 12181 raft_consensus.cc:2454] T ee7d49b432fb4c9783338e812dbd5354 P ada1bde3127445c2a898f7d7a85a157a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6c6104cd9c24ce7a0e04f247fc13ccb in term 0.
I20260927 21:29:03.297750 11904 raft_consensus.cc:2454] T ee7d49b432fb4c9783338e812dbd5354 P d76669faa6a348a39ce75b96675f9382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6c6104cd9c24ce7a0e04f247fc13ccb in term 0.
I20260927 21:29:03.298520 11980 leader_election.cc:302] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: ada1bde3127445c2a898f7d7a85a157a, d76669faa6a348a39ce75b96675f9382, f6c6104cd9c24ce7a0e04f247fc13ccb; no voters:
I20260927 21:29:03.299033 12253 raft_consensus.cc:2781] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:29:03.299188 12253 raft_consensus.cc:482] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:29:03.299299 12253 raft_consensus.cc:3037] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:29:03.301436 12253 raft_consensus.cc:504] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d76669faa6a348a39ce75b96675f9382" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40555 } } peers { permanent_uuid: "ada1bde3127445c2a898f7d7a85a157a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44771 } } peers { permanent_uuid: "f6c6104cd9c24ce7a0e04f247fc13ccb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } }
I20260927 21:29:03.302109 12253 leader_election.cc:288] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb [CANDIDATE]: Term 1 election: Requested vote from peers d76669faa6a348a39ce75b96675f9382 (127.0.97.1:40555), ada1bde3127445c2a898f7d7a85a157a (127.0.97.3:44771)
I20260927 21:29:03.302609 11904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee7d49b432fb4c9783338e812dbd5354" candidate_uuid: "f6c6104cd9c24ce7a0e04f247fc13ccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d76669faa6a348a39ce75b96675f9382"
I20260927 21:29:03.302644 12181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee7d49b432fb4c9783338e812dbd5354" candidate_uuid: "f6c6104cd9c24ce7a0e04f247fc13ccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada1bde3127445c2a898f7d7a85a157a"
I20260927 21:29:03.302820 11904 raft_consensus.cc:3037] T ee7d49b432fb4c9783338e812dbd5354 P d76669faa6a348a39ce75b96675f9382 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:29:03.302851 12181 raft_consensus.cc:3037] T ee7d49b432fb4c9783338e812dbd5354 P ada1bde3127445c2a898f7d7a85a157a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:29:03.305052 11904 raft_consensus.cc:2454] T ee7d49b432fb4c9783338e812dbd5354 P d76669faa6a348a39ce75b96675f9382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6c6104cd9c24ce7a0e04f247fc13ccb in term 1.
I20260927 21:29:03.305507 12181 raft_consensus.cc:2454] T ee7d49b432fb4c9783338e812dbd5354 P ada1bde3127445c2a898f7d7a85a157a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6c6104cd9c24ce7a0e04f247fc13ccb in term 1.
I20260927 21:29:03.305516 11981 leader_election.cc:302] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d76669faa6a348a39ce75b96675f9382, f6c6104cd9c24ce7a0e04f247fc13ccb; no voters:
I20260927 21:29:03.305927 12253 raft_consensus.cc:2781] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:29:03.306363 12253 raft_consensus.cc:686] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb [term 1 LEADER]: Becoming Leader. State: Replica: f6c6104cd9c24ce7a0e04f247fc13ccb, State: Running, Role: LEADER
I20260927 21:29:03.306901 12253 consensus_queue.cc:237] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d76669faa6a348a39ce75b96675f9382" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40555 } } peers { permanent_uuid: "ada1bde3127445c2a898f7d7a85a157a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44771 } } peers { permanent_uuid: "f6c6104cd9c24ce7a0e04f247fc13ccb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } }
I20260927 21:29:03.311311 11735 catalog_manager.cc:5495] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb reported cstate change: term changed from 0 to 1, leader changed from <none> to f6c6104cd9c24ce7a0e04f247fc13ccb (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f6c6104cd9c24ce7a0e04f247fc13ccb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d76669faa6a348a39ce75b96675f9382" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ada1bde3127445c2a898f7d7a85a157a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6c6104cd9c24ce7a0e04f247fc13ccb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36219 } health_report { overall_health: HEALTHY } } }
W20260927 21:29:03.335807 12227 tablet.cc:2369] T ee7d49b432fb4c9783338e812dbd5354 P ada1bde3127445c2a898f7d7a85a157a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:29:03.356153 12244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:03.350879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60042 (local address 127.0.97.2:36219)
0927 21:29:03.351145 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:03.351151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:03.351192 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:29:03.351351 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:03.351356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:03.351434 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:03.351585 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:29:03.351905 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.352622 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.354057 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.354478 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.354583 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:03.355047 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:29:03.355604 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:29:03.355730 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:03.355772 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 21:29:03.355838 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 21:29:03.364207 11904 raft_consensus.cc:1261] T ee7d49b432fb4c9783338e812dbd5354 P d76669faa6a348a39ce75b96675f9382 [term 1 FOLLOWER]: Refusing update from remote peer f6c6104cd9c24ce7a0e04f247fc13ccb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:29:03.364210 12181 raft_consensus.cc:1261] T ee7d49b432fb4c9783338e812dbd5354 P ada1bde3127445c2a898f7d7a85a157a [term 1 FOLLOWER]: Refusing update from remote peer f6c6104cd9c24ce7a0e04f247fc13ccb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:29:03.365214 12256 consensus_queue.cc:1035] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d76669faa6a348a39ce75b96675f9382" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:29:03.365567 12253 consensus_queue.cc:1035] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada1bde3127445c2a898f7d7a85a157a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:29:03.393700 11736 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:29:03.394045 11736 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:29:03.397854 11736 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82557e490bee40778605684c3f4af8cd: Sending DeleteTablet for 3 replicas of tablet ee7d49b432fb4c9783338e812dbd5354
I20260927 21:29:03.399016 12161 tablet_service.cc:1514] Processing DeleteTablet for tablet ee7d49b432fb4c9783338e812dbd5354 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:29:03 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49394
I20260927 21:29:03.399024 12027 tablet_service.cc:1514] Processing DeleteTablet for tablet ee7d49b432fb4c9783338e812dbd5354 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:29:03 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60036
I20260927 21:29:03.399003 11884 tablet_service.cc:1514] Processing DeleteTablet for tablet ee7d49b432fb4c9783338e812dbd5354 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:29:03 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59440
I20260927 21:29:03.402614 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 11816
I20260927 21:29:03.413805 12271 tablet_replica.cc:330] T ee7d49b432fb4c9783338e812dbd5354 P ada1bde3127445c2a898f7d7a85a157a: stopping tablet replica
I20260927 21:29:03.414106 12269 tablet_replica.cc:330] T ee7d49b432fb4c9783338e812dbd5354 P d76669faa6a348a39ce75b96675f9382: stopping tablet replica
I20260927 21:29:03.414497 12271 raft_consensus.cc:2229] T ee7d49b432fb4c9783338e812dbd5354 P ada1bde3127445c2a898f7d7a85a157a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:29:03.414520 12269 raft_consensus.cc:2229] T ee7d49b432fb4c9783338e812dbd5354 P d76669faa6a348a39ce75b96675f9382 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:29:03.415199 12269 raft_consensus.cc:2258] T ee7d49b432fb4c9783338e812dbd5354 P d76669faa6a348a39ce75b96675f9382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:29:03.415983 12270 tablet_replica.cc:330] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb: stopping tablet replica
I20260927 21:29:03.416415 12270 raft_consensus.cc:2229] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:29:03.417202 12270 raft_consensus.cc:2258] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:29:03.417984 12271 raft_consensus.cc:2258] T ee7d49b432fb4c9783338e812dbd5354 P ada1bde3127445c2a898f7d7a85a157a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:29:03.418666 12269 ts_tablet_manager.cc:1905] T ee7d49b432fb4c9783338e812dbd5354 P d76669faa6a348a39ce75b96675f9382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:29:03.420475 12271 ts_tablet_manager.cc:1905] T ee7d49b432fb4c9783338e812dbd5354 P ada1bde3127445c2a898f7d7a85a157a: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 21:29:03 dist-test-slave-bxm0 krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 21:29:03.426604 12270 ts_tablet_manager.cc:1905] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:29:03.428277 12271 ts_tablet_manager.cc:1918] T ee7d49b432fb4c9783338e812dbd5354 P ada1bde3127445c2a898f7d7a85a157a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:29:03.428434 12271 log.cc:1196] T ee7d49b432fb4c9783338e812dbd5354 P ada1bde3127445c2a898f7d7a85a157a: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-2/wal/wals/ee7d49b432fb4c9783338e812dbd5354
I20260927 21:29:03.428933 12271 ts_tablet_manager.cc:1939] T ee7d49b432fb4c9783338e812dbd5354 P ada1bde3127445c2a898f7d7a85a157a: Deleting consensus metadata
I20260927 21:29:03.430986 11720 catalog_manager.cc:4841] TS ada1bde3127445c2a898f7d7a85a157a (127.0.97.3:44771): tablet ee7d49b432fb4c9783338e812dbd5354 (table test-table [id=fb7eb2ec82c546dda517310f4881b6ea]) successfully deleted
I20260927 21:29:03.432514 12270 ts_tablet_manager.cc:1918] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:29:03.432705 12270 log.cc:1196] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-1/wal/wals/ee7d49b432fb4c9783338e812dbd5354
I20260927 21:29:03.433247 12270 ts_tablet_manager.cc:1939] T ee7d49b432fb4c9783338e812dbd5354 P f6c6104cd9c24ce7a0e04f247fc13ccb: Deleting consensus metadata
I20260927 21:29:03.435142 11723 catalog_manager.cc:4841] TS f6c6104cd9c24ce7a0e04f247fc13ccb (127.0.97.2:36219): tablet ee7d49b432fb4c9783338e812dbd5354 (table test-table [id=fb7eb2ec82c546dda517310f4881b6ea]) successfully deleted
I20260927 21:29:03.435616 12269 ts_tablet_manager.cc:1918] T ee7d49b432fb4c9783338e812dbd5354 P d76669faa6a348a39ce75b96675f9382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:29:03.435803 12269 log.cc:1196] T ee7d49b432fb4c9783338e812dbd5354 P d76669faa6a348a39ce75b96675f9382: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790544390748963-388-0/minicluster-data/ts-0/wal/wals/ee7d49b432fb4c9783338e812dbd5354
I20260927 21:29:03.436352 12269 ts_tablet_manager.cc:1939] T ee7d49b432fb4c9783338e812dbd5354 P d76669faa6a348a39ce75b96675f9382: Deleting consensus metadata
I20260927 21:29:03.438048 11721 catalog_manager.cc:4841] TS d76669faa6a348a39ce75b96675f9382 (127.0.97.1:40555): tablet ee7d49b432fb4c9783338e812dbd5354 (table test-table [id=fb7eb2ec82c546dda517310f4881b6ea]) successfully deleted
I20260927 21:29:03.440934 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:03.404093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59450 (local address 127.0.97.1:40555)
0927 21:29:03.404279 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:03.404285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:03.407949 (+ 3664us) server_negotiation.cc:407] Connection header received
0927 21:29:03.409063 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:03.409072 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:03.409223 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:03.409325 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:29:03.414791 (+ 5466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.415858 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.418372 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.418969 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.419085 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:03.425014 (+ 5929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:29:03.425030 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:29:03.425048 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:29:03.425095 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:29:03.437140 (+ 12045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:03.438527 (+ 1387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:03.438536 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:03.438540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:03.438707 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:03.439109 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:03.439114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:03.439118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:03.439575 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:29:03.439869 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:03.440264 (+ 395us) server_negotiation.cc:299] Negotiation successful
0927 21:29:03.440558 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:29:03.441677 11944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:29:03.548130 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 11816
I20260927 21:29:03.565820 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 11955
Sep 27 21:29:03 dist-test-slave-bxm0 krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 21:29:03.583118 12244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:03.566665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60048 (local address 127.0.97.2:36219)
0927 21:29:03.566855 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:03.566862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:03.566885 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:29:03.567057 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:03.567063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:03.567149 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:03.567262 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:29:03.568669 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.569950 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.571199 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.571852 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.571990 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:03.576185 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:29:03.576202 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:29:03.576223 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:29:03.576269 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:29:03.580224 (+ 3955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:03.580963 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:03.580973 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:03.580977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:03.581144 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:03.581500 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:03.581505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:03.581508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:03.581972 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:29:03.582304 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:03.582489 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 21:29:03.582706 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 21:29:03.583920 12087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:29:03.708982 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 11955
I20260927 21:29:03.727564 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 12098
Sep 27 21:29:03 dist-test-slave-bxm0 krb5kdc[11260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 21:29:03.743345 12242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:03.728430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49398 (local address 127.0.97.3:44771)
0927 21:29:03.728612 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:03.728620 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:03.728738 (+ 118us) server_negotiation.cc:407] Connection header received
0927 21:29:03.728940 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:03.728945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:03.729026 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:03.729132 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:29:03.730466 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.731203 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.732522 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:03.732920 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:03.733057 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:03.737580 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:29:03.737594 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:29:03.737659 (+ 65us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:29:03.737710 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:29:03.740578 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:03.741305 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:03.741313 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:03.741315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:03.741465 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:03.741897 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:03.741905 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:03.741911 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:03.742359 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:29:03.742611 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:03.742897 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 21:29:03.743071 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:29:03.743997 12221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:29:03.834591 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 12098
I20260927 21:29:03.850975 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 11704
I20260927 21:29:03.852272 11766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:29:03.962909 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 11704
I20260927 21:29:03.984925 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 21:29:04.330976 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 21:29:04.331235 388 mini_ranger.cc:66] Stopping Ranger...
I20260927 21:29:04.677284 388 mini_ranger.cc:68] Stopped Ranger
2026-09-27 21:29:04.677 UTC [11290] LOG: received smart shutdown request
2026-09-27 21:29:04.681 UTC [11290] LOG: background worker "logical replication launcher" (PID 11299) exited with exit code 1
2026-09-27 21:29:04.681 UTC [11294] LOG: shutting down
2026-09-27 21:29:04.843 UTC [11290] LOG: database system is shut down
2026-09-27T21:29:04Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50839 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:29:05 dist-test-slave-bxm0 krb5kdc[12285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:29:05 dist-test-slave-bxm0 krb5kdc[12285](info): set up 2 sockets
Sep 27 21:29:05 dist-test-slave-bxm0 krb5kdc[12285](info): commencing operation
krb5kdc: starting...
W20260927 21:29:06.094009 388 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:29:06 dist-test-slave-bxm0 krb5kdc[12285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:29:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:29:06Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260927 21:29:06.422466 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasklB5kC7/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/postgres -l logfile start
2026-09-27 21:29:08.995 UTC [12315] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 21:29:08.995 UTC [12315] LOG: listening on IPv4 address "127.0.97.20", port 34031
2026-09-27 21:29:08.999 UTC [12315] LOG: listening on Unix socket "/tmp/.s.PGSQL.34031"
2026-09-27 21:29:09.021 UTC [12318] LOG: database system was shut down at 2026-09-27 21:29:07 UTC
2026-09-27 21:29:09.025 UTC [12315] LOG: database system is ready to accept connections
I20260927 21:29:10.009704 388 mini_postgres.cc:96] Postgres bound to 34031
2026-09-27 21:29:10.026 UTC [12326] FATAL: database "slave" does not exist
127.0.97.20:34031 - accepting connections
I20260927 21:29:10.027310 388 mini_ranger.cc:161] Starting Ranger...
I20260927 21:29:10.053555 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 21:29:10.557778 388 mini_ranger.cc:87] Created ranger Postgres database
I20260927 21:29:10.557905 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ranger-admin
2026-09-27 21:29:10,854 [I] DB FLAVOR :POSTGRES
2026-09-27 21:29:10,855 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 21:29:10,855 [I] Checking connection..
2026-09-27 21:29:10,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 21:29:11,109 [I] Checking connection passed.
2026-09-27 21:29:11,110 [I] --------- Verifying version history table ---------
2026-09-27 21:29:11,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:29:11,360 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 21:29:11,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:29:11,615 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 21:29:11,616 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 21:29:11,616 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34031/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 21:29:11.877 UTC [12390] WARNING: there is no transaction in progress
2026-09-27 21:29:11,883 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 21:29:11,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:29:12,135 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 21:29:12,135 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 21:29:12,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 21:29:12,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 21:29:12,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-bxm0.c.gcp-upstream.internal','N') ;"
2026-09-27 21:29:12,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 21:29:13,117 [I] Table x_portal_user does not exist in database ranger
2026-09-27 21:29:13,117 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 21:29:13,359 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 21:29:13,359 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 21:29:13,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 21:29:13,862 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 21:29:13,863 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34031/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 21:29:14.789 UTC [12498] WARNING: there is no transaction in progress
2026-09-27 21:29:14.805 UTC [12498] WARNING: there is no transaction in progress
2026-09-27 21:29:14.821 UTC [12498] WARNING: there is no transaction in progress
2026-09-27 21:29:14.836 UTC [12498] WARNING: there is no transaction in progress
2026-09-27 21:29:14.852 UTC [12498] WARNING: there is no transaction in progress
2026-09-27 21:29:14.871 UTC [12498] WARNING: there is no transaction in progress
2026-09-27 21:29:14.925 UTC [12498] WARNING: there is no transaction in progress
2026-09-27 21:29:14.932 UTC [12498] WARNING: there is no transaction in progress
2026-09-27 21:29:14.946 UTC [12498] WARNING: there is no transaction in progress
2026-09-27 21:29:14.956 UTC [12498] WARNING: there is no transaction in progress
2026-09-27 21:29:14.966 UTC [12498] WARNING: there is no transaction in progress
2026-09-27 21:29:14.981 UTC [12498] WARNING: there is no transaction in progress
2026-09-27 21:29:14.990 UTC [12498] WARNING: there is no transaction in progress
2026-09-27 21:29:14.997 UTC [12498] WARNING: there is no transaction in progress
2026-09-27 21:29:15.004 UTC [12498] WARNING: there is no transaction in progress
2026-09-27 21:29:15.010 UTC [12498] WARNING: there is no transaction in progress
2026-09-27 21:29:15,641 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 21:29:15,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 21:29:15,863 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 21:29:15,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 21:29:16,096 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 21:29:16,096 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 21:29:16,327 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 21:29:16,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 21:29:16,586 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 21:29:16,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-bxm0.c.gcp-upstream.internal';"
2026-09-27 21:29:17,069 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 21:29:17,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34031/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 21:29:17,302 [I] DB_PATCHES have already been applied
I20260927 21:29:17.308203 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ranger-admin:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasklB5kC7/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 21:29:17.308287 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260927 21:29:17.322536 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:59557
Sep 27, 2026 9:29:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 9:29:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 9:29:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 9:29:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 9:29:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 9:29:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 9:29:18 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 21:29:18.476503 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.449s user 0.007s sys 0.031s
Sep 27, 2026 9:29:38 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 9:29:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 27, 2026 9:29:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 9:29:39 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 9:29:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 9:29:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 9:29:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 9:29:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 9:29:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 9:29:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 9:29:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 9:29:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 9:29:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 9:29:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 9:29:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 9:29:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 9:29:39 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 9:29:39 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 21:29:40.580034 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 21:29:40 dist-test-slave-bxm0 krb5kdc[12285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544580, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 21:29:40.756420 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 21:29:40.779641 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 21:29:41.700603 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 21:29:41.700722 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ranger-kms
I20260927 21:29:41.700774 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:34031
2026-09-27 21:29:42,054 [I] DB FLAVOR :POSTGRES
2026-09-27 21:29:42,054 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 21:29:42,054 [I] Checking connection
2026-09-27 21:29:42,054 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34031/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 21:29:42,286 [I] connection success
2026-09-27 21:29:42,286 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 21:29:42,287 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 21:29:42,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34031/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 21:29:42,533 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 21:29:42,533 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 21:29:42,534 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 21:29:42,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:34031/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 21:29:42,792 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 21:29:43.278878 388 mini_ranger_kms.cc:310] Created kms service
I20260927 21:29:43.453544 388 mini_ranger_kms.cc:326] Created kudu user
I20260927 21:29:43.497179 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 21:29:43 dist-test-slave-bxm0 krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544580, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 21:29:43.874910 388 mini_ranger_kms.cc:384] Added ranger policy
I20260927 21:29:43.875331 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ranger-kms:/tmp/dist-test-tasklB5kC7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasklB5kC7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 21:29:43.875418 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260927 21:29:43.890390 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12678
I20260927 21:29:43.890537 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:45085
Sep 27, 2026 9:29:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 9:29:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 9:29:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 9:29:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 9:29:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 9:29:45 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 21:29:46.201573 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.445s user 0.070s sys 0.301s
I20260927 21:29:46.201876 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 21:29:46.202067 388 mini_ranger_kms.cc:401] 127.0.97.20:45085/kms/v1/keys
Sep 27 21:29:46 dist-test-slave-bxm0 krb5kdc[12285](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790544586, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 21:29:46 dist-test-slave-bxm0 krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544586, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 21:29:47 dist-test-slave-bxm0 krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544586, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 21:29:49 dist-test-slave-bxm0 krb5kdc[12285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:29:49.247170 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45961
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45961
--ranger_config_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45085/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:29:49.597743 12729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:29:49.598027 12729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:29:49.598119 12729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:29:49.605268 12729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:29:49.605368 12729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:29:49.605412 12729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:29:49.605474 12729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:29:49.605523 12729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:29:49.615505 12729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:45085/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45961
--ranger_config_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:29:49.616696 12729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:29:49.617988 12729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:29:49.631618 12734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:29:49.631531 12737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:29:49.631553 12729 server_base.cc:1034] running on GCE node
W20260927 21:29:49.631665 12735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:29:49.632870 12729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:29:49.634295 12729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:29:49.635510 12729 hybrid_clock.cc:648] HybridClock initialized: now 1790544589635463 us; error 53 us; skew 500 ppm
I20260927 21:29:49.635877 12729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:29:49 dist-test-slave-bxm0 krb5kdc[12285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:29:49.638808 12729 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:29:49.640130 12729 webserver.cc:460] Webserver started at http://127.0.97.62:36525/ using document root <none> and password file <none>
I20260927 21:29:49.640729 12729 fs_manager.cc:359] Metadata directory not provided
I20260927 21:29:49.640820 12729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:29:49.641059 12729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:29:49 dist-test-slave-bxm0 krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 21:29:49.698232 12729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "2ae8cda3649b4706829f4fbf77708129"
format_stamp: "Formatted at 2026-09-27 21:29:49 on dist-test-slave-bxm0"
server_key: "5de7a5b55553330cd4377e56885cb47a"
server_key_iv: "c7e4881573722dd37fc1c4a8aa8aec72"
server_key_version: "kuduclusterkey@0"
I20260927 21:29:49.699035 12729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "2ae8cda3649b4706829f4fbf77708129"
format_stamp: "Formatted at 2026-09-27 21:29:49 on dist-test-slave-bxm0"
server_key: "5de7a5b55553330cd4377e56885cb47a"
server_key_iv: "c7e4881573722dd37fc1c4a8aa8aec72"
server_key_version: "kuduclusterkey@0"
I20260927 21:29:49.704252 12729 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:29:49.720813 12744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:29:49.722421 12729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 21:29:49.722625 12729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "2ae8cda3649b4706829f4fbf77708129"
format_stamp: "Formatted at 2026-09-27 21:29:49 on dist-test-slave-bxm0"
server_key: "5de7a5b55553330cd4377e56885cb47a"
server_key_iv: "c7e4881573722dd37fc1c4a8aa8aec72"
server_key_version: "kuduclusterkey@0"
I20260927 21:29:49.722856 12729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:29:49.743732 12729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:29:49.747819 12729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:29:49.748129 12729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:29:49.766263 12729 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45961
I20260927 21:29:49.766299 12795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45961 every 8 connection(s)
I20260927 21:29:49.767683 12729 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:29:49.770815 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 12729
I20260927 21:29:49.771019 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:29:49.771827 12796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:29:49.771844 388 external_mini_cluster.cc:1428] Setting key 77cd8f9f7f791926fe1d547ca2769e50
Sep 27 21:29:49 dist-test-slave-bxm0 krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:29:49.786315 12796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129: Bootstrap starting.
I20260927 21:29:49.790509 12796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129: Neither blocks nor log segments found. Creating new log.
I20260927 21:29:49.791832 12796 log.cc:824] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129: Log is configured to *not* fsync() on all Append() calls
I20260927 21:29:49.794911 12796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129: No bootstrap required, opened a new log
I20260927 21:29:49.795604 12799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:49.773886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41920 (local address 127.0.97.62:45961)
0927 21:29:49.774669 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:49.774683 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:49.774724 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:29:49.776510 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:49.776532 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:49.776917 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:49.777252 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:29:49.778941 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:49.780207 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:49.781998 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:49.782667 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:49.782812 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:49.787360 (+ 4548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:29:49.787373 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:29:49.787392 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:29:49.787459 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:29:49.791363 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:49.792156 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:49.792164 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:49.792171 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:49.792337 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:49.792745 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:49.792749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:49.792751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:49.793330 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:29:49.793691 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:49.794124 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 21:29:49.794521 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":354,"threads_started":1}
I20260927 21:29:49.802865 12796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae8cda3649b4706829f4fbf77708129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45961 } }
I20260927 21:29:49.803246 12796 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:29:49.803360 12796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae8cda3649b4706829f4fbf77708129, State: Initialized, Role: FOLLOWER
I20260927 21:29:49.803901 12796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae8cda3649b4706829f4fbf77708129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45961 } }
I20260927 21:29:49.804147 12796 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:29:49.804293 12796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:29:49.804456 12796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:29:49.806846 12796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae8cda3649b4706829f4fbf77708129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45961 } }
I20260927 21:29:49.807376 12796 leader_election.cc:302] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ae8cda3649b4706829f4fbf77708129; no voters:
I20260927 21:29:49.807927 12796 leader_election.cc:288] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:29:49.808080 12801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:29:49.808768 12801 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [term 1 LEADER]: Becoming Leader. State: Replica: 2ae8cda3649b4706829f4fbf77708129, State: Running, Role: LEADER
I20260927 21:29:49.809296 12801 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae8cda3649b4706829f4fbf77708129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45961 } }
I20260927 21:29:49.810105 12796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:29:49.812507 12803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ae8cda3649b4706829f4fbf77708129. Latest consensus state: current_term: 1 leader_uuid: "2ae8cda3649b4706829f4fbf77708129" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae8cda3649b4706829f4fbf77708129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45961 } } }
I20260927 21:29:49.812963 12803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [sys.catalog]: This master's current role is: LEADER
I20260927 21:29:49.812707 12802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ae8cda3649b4706829f4fbf77708129" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae8cda3649b4706829f4fbf77708129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45961 } } }
I20260927 21:29:49.813198 12802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [sys.catalog]: This master's current role is: LEADER
I20260927 21:29:49.814385 12809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:29:49.820032 12809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:29:49.820607 12796 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 21:29:49.838328 12809 catalog_manager.cc:1347] Generated new cluster ID: 66199bb7d2da4b1788e566b3273f2aca
I20260927 21:29:49.838502 12809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:29:49.873924 12809 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:29:49.876026 12809 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:29:49.884859 12809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129: Generated new TSK 0
I20260927 21:29:49.885999 12809 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 21:29:50.577116 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45085/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45961
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:29:50 dist-test-slave-bxm0 krb5kdc[12285](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790544590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 21:29:51.016438 12841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:29:51.016739 12841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:29:51.016909 12841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:29:51.024291 12841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:29:51.024435 12841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:29:51.024619 12841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:29:51.036764 12841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:45085/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:29:51.038349 12841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:29:51.039772 12841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:29:51.055528 12851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:29:51.056041 12852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:29:51.055725 12854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:29:51.057166 12841 server_base.cc:1034] running on GCE node
I20260927 21:29:51.057873 12841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:29:51.058848 12841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:29:51.060075 12841 hybrid_clock.cc:648] HybridClock initialized: now 1790544591060017 us; error 55 us; skew 500 ppm
I20260927 21:29:51.060389 12841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:29:51 dist-test-slave-bxm0 krb5kdc[12285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:29:51.063745 12841 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 21:29:51.065330 12841 webserver.cc:460] Webserver started at http://127.0.97.1:44565/ using document root <none> and password file <none>
I20260927 21:29:51.066074 12841 fs_manager.cc:359] Metadata directory not provided
I20260927 21:29:51.066210 12841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:29:51.066605 12841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:29:51 dist-test-slave-bxm0 krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 21:29:51.086690 12841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/data/instance:
uuid: "8dafde60964d43b2b14b4ff7a5a9fb37"
format_stamp: "Formatted at 2026-09-27 21:29:51 on dist-test-slave-bxm0"
server_key: "d13a2b16691f622abe870489b2028cef"
server_key_iv: "e2cfea7f996d44a414ba6f4d706631a5"
server_key_version: "kuduclusterkey@0"
I20260927 21:29:51.087924 12841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8dafde60964d43b2b14b4ff7a5a9fb37"
format_stamp: "Formatted at 2026-09-27 21:29:51 on dist-test-slave-bxm0"
server_key: "d13a2b16691f622abe870489b2028cef"
server_key_iv: "e2cfea7f996d44a414ba6f4d706631a5"
server_key_version: "kuduclusterkey@0"
I20260927 21:29:51.095397 12841 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260927 21:29:51.131177 12861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:29:51.133253 12841 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.009s sys 0.003s
I20260927 21:29:51.133467 12841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "8dafde60964d43b2b14b4ff7a5a9fb37"
format_stamp: "Formatted at 2026-09-27 21:29:51 on dist-test-slave-bxm0"
server_key: "d13a2b16691f622abe870489b2028cef"
server_key_iv: "e2cfea7f996d44a414ba6f4d706631a5"
server_key_version: "kuduclusterkey@0"
I20260927 21:29:51.133797 12841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:29:51.155865 12841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:29:51.161890 12841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:29:51.162254 12841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:29:51.163471 12841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:29:51.165297 12841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:29:51.165395 12841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:29:51.165582 12841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:29:51.165719 12841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:29:51.223964 12841 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41611
I20260927 21:29:51.223984 12973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41611 every 8 connection(s)
I20260927 21:29:51.225243 12841 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:29:51.228046 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 12841
I20260927 21:29:51.228242 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/wal/instance
I20260927 21:29:51.229105 388 external_mini_cluster.cc:1428] Setting key fb10013c4335480094ad2ea39828a6c5
Sep 27 21:29:51 dist-test-slave-bxm0 krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:29:51.274015 12976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:51.228709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51531 (local address 127.0.97.62:45961)
0927 21:29:51.249309 (+ 20600us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:51.249313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:51.251422 (+ 2109us) server_negotiation.cc:407] Connection header received
0927 21:29:51.254338 (+ 2916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:51.254346 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:51.254454 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:51.254542 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:29:51.257024 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:51.258229 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:51.260740 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:51.261382 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:51.261486 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:51.266276 (+ 4790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:29:51.266283 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:29:51.266288 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:29:51.266340 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:29:51.269971 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:51.271282 (+ 1311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:51.271290 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:51.271294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:51.271434 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:51.272003 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:51.272008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:51.272024 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:51.272298 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:29:51.272450 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:51.273364 (+ 914us) server_negotiation.cc:299] Negotiation successful
0927 21:29:51.273581 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20468,"thread_start_us":20347,"threads_started":1}
I20260927 21:29:51.275271 12977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:51.229288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45961 (local address 127.0.97.1:51531)
0927 21:29:51.251215 (+ 21927us) negotiation.cc:107] Waiting for socket to connect
0927 21:29:51.251276 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 21:29:51.253861 (+ 2585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:29:51.255203 (+ 1342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:29:51.255217 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:29:51.255900 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:29:51.256784 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:51.256810 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:51.258450 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:51.258456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:51.260553 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:51.260568 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:51.261867 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:51.261875 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:51.262187 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:51.263067 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:29:51.263105 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:29:51.265994 (+ 2889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:29:51.270434 (+ 4440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:29:51.270446 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:29:51.270468 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:29:51.271156 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:29:51.271567 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:29:51.271574 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:29:51.271580 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:29:51.271880 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:29:51.272440 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:29:51.272448 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:29:51.272949 (+ 501us) client_negotiation.cc:769] Sending connection context
0927 21:29:51.273288 (+ 339us) client_negotiation.cc:240] Negotiation successful
0927 21:29:51.273848 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21559,"thread_start_us":33374,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 21:29:51.277271 12974 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45961
I20260927 21:29:51.277570 12974 heartbeater.cc:464] Registering TS with master...
I20260927 21:29:51.278281 12974 heartbeater.cc:511] Master 127.0.97.62:45961 requested a full tablet report, sending...
I20260927 21:29:51.280678 12761 ts_manager.cc:194] Registered new tserver with Master: 8dafde60964d43b2b14b4ff7a5a9fb37 (127.0.97.1:41611)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 21:29:51.282766 12761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 21:29:51.311347 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45085/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45961
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:29:51 dist-test-slave-bxm0 krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 21:29:51 dist-test-slave-bxm0 krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260927 21:29:51.739951 12980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:29:51.740460 12980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:29:51.740666 12980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:29:51.751319 12980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:29:51.751462 12980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:29:51.751660 12980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:29:51.767059 12980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:45085/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:29:51.769044 12980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:29:51.770852 12980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:29:51.789870 12987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:29:51.789870 12988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:29:51.795857 12990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:29:51.796114 12980 server_base.cc:1034] running on GCE node
I20260927 21:29:51.796937 12980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:29:51.799835 12980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:29:51.801123 12980 hybrid_clock.cc:648] HybridClock initialized: now 1790544591801019 us; error 101 us; skew 500 ppm
I20260927 21:29:51.801522 12980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:29:51 dist-test-slave-bxm0 krb5kdc[12285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:29:51.807443 12980 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 21:29:51.810190 12980 webserver.cc:460] Webserver started at http://127.0.97.2:36273/ using document root <none> and password file <none>
I20260927 21:29:51.811136 12980 fs_manager.cc:359] Metadata directory not provided
I20260927 21:29:51.811275 12980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:29:51.811609 12980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:29:51 dist-test-slave-bxm0 krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 21:29:51.835778 12980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/data/instance:
uuid: "1134e53ebae94f2bafdd4709c20c4b27"
format_stamp: "Formatted at 2026-09-27 21:29:51 on dist-test-slave-bxm0"
server_key: "11483d4df982401cb14a50ea6cf66797"
server_key_iv: "d29781be021a1ab2d04e4de6c5beca32"
server_key_version: "kuduclusterkey@0"
I20260927 21:29:51.837225 12980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1134e53ebae94f2bafdd4709c20c4b27"
format_stamp: "Formatted at 2026-09-27 21:29:51 on dist-test-slave-bxm0"
server_key: "11483d4df982401cb14a50ea6cf66797"
server_key_iv: "d29781be021a1ab2d04e4de6c5beca32"
server_key_version: "kuduclusterkey@0"
I20260927 21:29:51.847873 12980 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.002s sys 0.006s
I20260927 21:29:51.890261 12997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:29:51.892305 12980 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.006s
I20260927 21:29:51.892555 12980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "1134e53ebae94f2bafdd4709c20c4b27"
format_stamp: "Formatted at 2026-09-27 21:29:51 on dist-test-slave-bxm0"
server_key: "11483d4df982401cb14a50ea6cf66797"
server_key_iv: "d29781be021a1ab2d04e4de6c5beca32"
server_key_version: "kuduclusterkey@0"
I20260927 21:29:51.892835 12980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:29:51.929579 12980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:29:51.935813 12980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:29:51.936235 12980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:29:51.937552 12980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:29:51.945956 12980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:29:51.946122 12980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:29:51.946302 12980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:29:51.946403 12980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:29:52.065047 12980 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33023
I20260927 21:29:52.065064 13114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33023 every 8 connection(s)
I20260927 21:29:52.066653 12980 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:29:52.072765 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 12980
I20260927 21:29:52.072964 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/wal/instance
I20260927 21:29:52.073565 388 external_mini_cluster.cc:1428] Setting key 3b621767d3a86a369b607ac046dc4dbd
Sep 27 21:29:52 dist-test-slave-bxm0 krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:29:52.090047 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:52.069300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41997 (local address 127.0.97.62:45961)
0927 21:29:52.069820 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:52.069825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:52.070869 (+ 1044us) server_negotiation.cc:407] Connection header received
0927 21:29:52.072980 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:52.072989 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:52.073115 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:52.073215 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:29:52.075034 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:52.075922 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:52.078595 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:52.079039 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:52.079158 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:52.083643 (+ 4485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:29:52.083649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:29:52.083652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:29:52.083714 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:29:52.086882 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:52.087663 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:52.087673 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:52.087675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:52.087805 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:52.088275 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:52.088279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:52.088281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:52.088544 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:29:52.088701 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:52.089396 (+ 695us) server_negotiation.cc:299] Negotiation successful
0927 21:29:52.089648 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":219,"threads_started":1}
I20260927 21:29:52.091050 13120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:52.069909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45961 (local address 127.0.97.2:41997)
0927 21:29:52.070677 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0927 21:29:52.070724 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:29:52.072612 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:29:52.073507 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:29:52.073519 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:29:52.074138 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:29:52.074864 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:52.074889 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:52.076086 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:52.076091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:52.078375 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:52.078393 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:52.079287 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:52.079296 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:52.079651 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:52.080411 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:29:52.080456 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:29:52.083360 (+ 2904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:29:52.087039 (+ 3679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:29:52.087048 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:29:52.087075 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:29:52.087518 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:29:52.087919 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:29:52.087924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:29:52.087929 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:29:52.088145 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:29:52.088681 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:29:52.088689 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:29:52.089026 (+ 337us) client_negotiation.cc:769] Sending connection context
0927 21:29:52.089297 (+ 271us) client_negotiation.cc:240] Negotiation successful
0927 21:29:52.089879 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":313,"threads_started":1}
I20260927 21:29:52.092545 13117 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45961
I20260927 21:29:52.092857 13117 heartbeater.cc:464] Registering TS with master...
I20260927 21:29:52.093566 13117 heartbeater.cc:511] Master 127.0.97.62:45961 requested a full tablet report, sending...
I20260927 21:29:52.095335 12761 ts_manager.cc:194] Registered new tserver with Master: 1134e53ebae94f2bafdd4709c20c4b27 (127.0.97.2:33023)
I20260927 21:29:52.096158 12761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41997
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 21:29:52.126688 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45085/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45961
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:29:52.287050 12974 heartbeater.cc:503] Master 127.0.97.62:45961 was elected leader, sending a full tablet report...
W20260927 21:29:52.456099 13123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:29:52.456378 13123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:29:52.456482 13123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:29:52.463861 13123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:29:52.463979 13123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:29:52.464180 13123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:29:52.475008 13123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:45085/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:29:52.476380 13123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:29:52.477819 13123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:29:52.491350 13128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:29:52.491324 13129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:29:52.491271 13131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:29:52.492254 13123 server_base.cc:1034] running on GCE node
I20260927 21:29:52.492899 13123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:29:52.493937 13123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:29:52.495165 13123 hybrid_clock.cc:648] HybridClock initialized: now 1790544592495107 us; error 52 us; skew 500 ppm
I20260927 21:29:52.495498 13123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:29:52 dist-test-slave-bxm0 krb5kdc[12285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544592, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:29:52.498606 13123 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 21:29:52.500342 13123 webserver.cc:460] Webserver started at http://127.0.97.3:43269/ using document root <none> and password file <none>
I20260927 21:29:52.501065 13123 fs_manager.cc:359] Metadata directory not provided
I20260927 21:29:52.501211 13123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:29:52.501525 13123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:29:52 dist-test-slave-bxm0 krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544592, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 21:29:52.518208 13123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/data/instance:
uuid: "0ebceed10feb4d1da01bfcd9a7ad1b1a"
format_stamp: "Formatted at 2026-09-27 21:29:52 on dist-test-slave-bxm0"
server_key: "8d81ce058256998461d914404c97d421"
server_key_iv: "334206cd928b81e53ab52386d7d3f082"
server_key_version: "kuduclusterkey@0"
I20260927 21:29:52.519304 13123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0ebceed10feb4d1da01bfcd9a7ad1b1a"
format_stamp: "Formatted at 2026-09-27 21:29:52 on dist-test-slave-bxm0"
server_key: "8d81ce058256998461d914404c97d421"
server_key_iv: "334206cd928b81e53ab52386d7d3f082"
server_key_version: "kuduclusterkey@0"
I20260927 21:29:52.526306 13123 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260927 21:29:52.542198 13138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:29:52.543752 13123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:29:52.543934 13123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "0ebceed10feb4d1da01bfcd9a7ad1b1a"
format_stamp: "Formatted at 2026-09-27 21:29:52 on dist-test-slave-bxm0"
server_key: "8d81ce058256998461d914404c97d421"
server_key_iv: "334206cd928b81e53ab52386d7d3f082"
server_key_version: "kuduclusterkey@0"
I20260927 21:29:52.544181 13123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:29:52.561877 13123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:29:52.566063 13123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:29:52.566447 13123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:29:52.567425 13123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:29:52.568856 13123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:29:52.568928 13123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:29:52.569084 13123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:29:52.569134 13123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:29:52.601985 13123 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36505
I20260927 21:29:52.602036 13250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36505 every 8 connection(s)
I20260927 21:29:52.603297 13123 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:29:52.609751 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 13123
I20260927 21:29:52.609939 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/wal/instance
I20260927 21:29:52.610432 388 external_mini_cluster.cc:1428] Setting key a7abe42fa87cb3ae4bf33e6a66bdfe0b
Sep 27 21:29:52 dist-test-slave-bxm0 krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544592, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:29:52.622951 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:52.605839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34597 (local address 127.0.97.62:45961)
0927 21:29:52.606363 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:52.606367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:52.607124 (+ 757us) server_negotiation.cc:407] Connection header received
0927 21:29:52.608877 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:52.608882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:52.608978 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:52.609102 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:29:52.610699 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:52.611696 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:52.613482 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:52.614006 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:52.614136 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:52.617575 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:29:52.617582 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:29:52.617588 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:29:52.617686 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:29:52.620092 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:52.620827 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:52.620833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:52.620836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:52.621007 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:52.621415 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:52.621418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:52.621420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:52.621773 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:29:52.621886 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:52.622521 (+ 635us) server_negotiation.cc:299] Negotiation successful
0927 21:29:52.622686 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":267,"threads_started":1}
I20260927 21:29:52.624043 13254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:52.606284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45961 (local address 127.0.97.3:34597)
0927 21:29:52.606968 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0927 21:29:52.607005 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:29:52.608616 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:29:52.609331 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:29:52.609339 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:29:52.609821 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:29:52.610504 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:52.610545 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:52.611828 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:52.611833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:52.613342 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:52.613355 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:52.614242 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:52.614251 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:52.614527 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:52.615199 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:29:52.615221 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:29:52.617338 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:29:52.620245 (+ 2907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:29:52.620254 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:29:52.620272 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:29:52.620698 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:29:52.621132 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:29:52.621135 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:29:52.621138 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:29:52.621315 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:29:52.621878 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:29:52.621884 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:29:52.622199 (+ 315us) client_negotiation.cc:769] Sending connection context
0927 21:29:52.622437 (+ 238us) client_negotiation.cc:240] Negotiation successful
0927 21:29:52.622817 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":278,"threads_started":1}
I20260927 21:29:52.625571 13251 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45961
I20260927 21:29:52.625854 13251 heartbeater.cc:464] Registering TS with master...
I20260927 21:29:52.626601 13251 heartbeater.cc:511] Master 127.0.97.62:45961 requested a full tablet report, sending...
I20260927 21:29:52.628026 12761 ts_manager.cc:194] Registered new tserver with Master: 0ebceed10feb4d1da01bfcd9a7ad1b1a (127.0.97.3:36505)
I20260927 21:29:52.628818 12761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34597
I20260927 21:29:52.638042 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:29:52.639688 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 12841
Sep 27 21:29:52 dist-test-slave-bxm0 krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 21:29:52.656589 13256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:52.640509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39894 (local address 127.0.97.1:41611)
0927 21:29:52.641099 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:52.641105 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:52.641131 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:29:52.641188 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:52.641194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:52.641398 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:52.641543 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:29:52.642904 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:52.643885 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:52.645417 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:52.646029 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:52.646129 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:52.650416 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:29:52.650430 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:29:52.650445 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:29:52.650478 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:29:52.653364 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:52.654081 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:52.654087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:52.654089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:52.654291 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:52.654784 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:52.654789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:52.654793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:52.655233 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:29:52.655549 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:52.655878 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 21:29:52.656162 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":294,"threads_started":1}
I20260927 21:29:52.657904 12969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:29:52.741015 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 12841
I20260927 21:29:52.757776 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 12980
Sep 27 21:29:52 dist-test-slave-bxm0 krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 21:29:52.773584 13258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:52.758674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50810 (local address 127.0.97.2:33023)
0927 21:29:52.759202 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:52.759209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:52.759233 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:29:52.759289 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:52.759296 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:52.759498 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:52.759618 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:29:52.760831 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:52.761873 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:52.763139 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:52.763647 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:52.763736 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:52.767835 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:29:52.767846 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:29:52.767860 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:29:52.767898 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:29:52.770735 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:52.771466 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:52.771483 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:52.771485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:52.771621 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:52.771996 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:52.772002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:52.772018 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:52.772349 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:29:52.772659 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:52.772960 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 21:29:52.773238 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":294,"threads_started":1}
I20260927 21:29:52.774884 13108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:29:52.856873 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 12980
I20260927 21:29:52.876595 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 13123
Sep 27 21:29:52 dist-test-slave-bxm0 krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 21:29:52.892441 13260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:52.877586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38144 (local address 127.0.97.3:36505)
0927 21:29:52.878234 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:52.878241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:52.878269 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:29:52.878329 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:52.878335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:52.878547 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:52.878672 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:29:52.879909 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:52.880896 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:52.882283 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:52.882812 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:52.882911 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:52.887005 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:29:52.887015 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:29:52.887030 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:29:52.887059 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:29:52.890004 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:52.890619 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:52.890625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:52.890627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:52.890742 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:52.891045 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:52.891048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:52.891050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:52.891360 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:29:52.891592 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:52.891848 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 21:29:52.892056 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":347,"threads_started":1}
I20260927 21:29:52.893764 13246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:29:52.976141 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 13123
I20260927 21:29:52.991716 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 12729
I20260927 21:29:52.992971 12791 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:29:53.098351 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 12729
I20260927 21:29:53.122010 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45961
--webserver_interface=127.0.97.62
--webserver_port=36525
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45961
--ranger_config_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45085/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45085/kms with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:29:53.448518 13263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:29:53.448810 13263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:29:53.448985 13263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:29:53.456360 13263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:29:53.456470 13263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:29:53.456514 13263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:29:53.456593 13263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:29:53.456679 13263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:29:53.467464 13263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45085/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45961
--ranger_config_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=36525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:29:53.468782 13263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:29:53.470181 13263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:29:53.482368 13269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:29:53.482443 13271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:29:53.482374 13268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:29:53.483395 13263 server_base.cc:1034] running on GCE node
I20260927 21:29:53.484169 13263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:29:53.485702 13263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:29:53.486923 13263 hybrid_clock.cc:648] HybridClock initialized: now 1790544593486869 us; error 49 us; skew 500 ppm
I20260927 21:29:53.487293 13263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:29:53 dist-test-slave-bxm0 krb5kdc[12285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:29:53.490530 13263 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:29:53.492079 13263 webserver.cc:460] Webserver started at http://127.0.97.62:36525/ using document root <none> and password file <none>
I20260927 21:29:53.492786 13263 fs_manager.cc:359] Metadata directory not provided
I20260927 21:29:53.492936 13263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 21:29:53 dist-test-slave-bxm0 krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 21:29:53.540949 13263 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:29:53.544014 13279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:29:53.545368 13263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:29:53.545598 13263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "2ae8cda3649b4706829f4fbf77708129"
format_stamp: "Formatted at 2026-09-27 21:29:49 on dist-test-slave-bxm0"
server_key: "5de7a5b55553330cd4377e56885cb47a"
server_key_iv: "c7e4881573722dd37fc1c4a8aa8aec72"
server_key_version: "kuduclusterkey@0"
I20260927 21:29:53.546658 13263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:29:53.562456 13263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:29:53.566156 13263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:29:53.566450 13263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:29:53.584669 13263 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45961
I20260927 21:29:53.584690 13330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45961 every 8 connection(s)
I20260927 21:29:53.586229 13263 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:29:53.593112 13331 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:29:53.594585 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 13263
I20260927 21:29:53.595623 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45085/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41611
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45961
--webserver_port=44565
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45085/kms with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:29:53.597877 13331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129: Bootstrap starting.
I20260927 21:29:53.618371 13331 log.cc:824] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129: Log is configured to *not* fsync() on all Append() calls
I20260927 21:29:53.630175 13331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:29:53.630862 13331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129: Bootstrap complete.
I20260927 21:29:53.640044 13331 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae8cda3649b4706829f4fbf77708129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45961 } }
I20260927 21:29:53.640877 13331 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae8cda3649b4706829f4fbf77708129, State: Initialized, Role: FOLLOWER
I20260927 21:29:53.641530 13331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae8cda3649b4706829f4fbf77708129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45961 } }
I20260927 21:29:53.641814 13331 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:29:53.641937 13331 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:29:53.642124 13331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:29:53.646979 13331 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae8cda3649b4706829f4fbf77708129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45961 } }
I20260927 21:29:53.647488 13331 leader_election.cc:302] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ae8cda3649b4706829f4fbf77708129; no voters:
I20260927 21:29:53.648119 13331 leader_election.cc:288] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 21:29:53.648259 13335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:29:53.650576 13331 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:29:53.655387 13335 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [term 2 LEADER]: Becoming Leader. State: Replica: 2ae8cda3649b4706829f4fbf77708129, State: Running, Role: LEADER
I20260927 21:29:53.656085 13335 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae8cda3649b4706829f4fbf77708129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45961 } }
I20260927 21:29:53.662997 13336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2ae8cda3649b4706829f4fbf77708129" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae8cda3649b4706829f4fbf77708129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45961 } } }
I20260927 21:29:53.663516 13336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [sys.catalog]: This master's current role is: LEADER
I20260927 21:29:53.664140 13337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ae8cda3649b4706829f4fbf77708129. Latest consensus state: current_term: 2 leader_uuid: "2ae8cda3649b4706829f4fbf77708129" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae8cda3649b4706829f4fbf77708129" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45961 } } }
I20260927 21:29:53.664476 13337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129 [sys.catalog]: This master's current role is: LEADER
I20260927 21:29:53.667326 13345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:29:53.673174 13345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:29:53.674026 13331 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 21:29:53.674783 13345 catalog_manager.cc:1259] Loaded cluster ID: 66199bb7d2da4b1788e566b3273f2aca
I20260927 21:29:53.674876 13345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:29:53.679160 13345 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:29:53.680668 13345 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129: Loaded TSK: 0
I20260927 21:29:53.682264 13345 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:29:54.031080 13333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:29:54.031551 13333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:29:54.031780 13333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:29:54.041256 13333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:29:54.041496 13333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:29:54.041771 13333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:29:54.058526 13333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45085/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41611
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44565
--tserver_master_addrs=127.0.97.62:45961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:29:54.060457 13333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:29:54.061952 13333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:29:54.082813 13370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:29:54.082809 13369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:29:54.090039 13372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:29:54.092134 13333 server_base.cc:1034] running on GCE node
I20260927 21:29:54.093091 13333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:29:54.106282 13333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:29:54.109800 13333 hybrid_clock.cc:648] HybridClock initialized: now 1790544594109696 us; error 100 us; skew 500 ppm
I20260927 21:29:54.110332 13333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:29:54 dist-test-slave-bxm0 krb5kdc[12285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:29:54.115146 13333 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 21:29:54.121932 13333 webserver.cc:460] Webserver started at http://127.0.97.1:44565/ using document root <none> and password file <none>
I20260927 21:29:54.123057 13333 fs_manager.cc:359] Metadata directory not provided
I20260927 21:29:54.123306 13333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 21:29:54 dist-test-slave-bxm0 krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 21:29:54.162384 13333 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.008s sys 0.003s
I20260927 21:29:54.168977 13380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:29:54.171088 13333 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.006s sys 0.000s
I20260927 21:29:54.171384 13333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "8dafde60964d43b2b14b4ff7a5a9fb37"
format_stamp: "Formatted at 2026-09-27 21:29:51 on dist-test-slave-bxm0"
server_key: "d13a2b16691f622abe870489b2028cef"
server_key_iv: "e2cfea7f996d44a414ba6f4d706631a5"
server_key_version: "kuduclusterkey@0"
I20260927 21:29:54.173084 13333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:29:54.205833 13333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:29:54.211733 13333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:29:54.212218 13333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:29:54.219859 13333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:29:54.228416 13333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:29:54.228634 13333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:29:54.228825 13333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:29:54.228989 13333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:29:54.296592 13333 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41611
I20260927 21:29:54.296653 13492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41611 every 8 connection(s)
I20260927 21:29:54.298698 13333 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:29:54.308727 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 13333
I20260927 21:29:54.309578 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45085/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33023
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45961
--webserver_port=36273
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45085/kms with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:29:54 dist-test-slave-bxm0 krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:29:54.348672 13497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:54.302924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45961 (local address 127.0.97.1:50373)
0927 21:29:54.321989 (+ 19065us) negotiation.cc:107] Waiting for socket to connect
0927 21:29:54.322089 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0927 21:29:54.323939 (+ 1850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:29:54.325868 (+ 1929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:29:54.325879 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:29:54.326485 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:29:54.327268 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:54.327305 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:54.329078 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:54.329083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:54.330456 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:54.330469 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:54.331639 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:54.331645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:54.331919 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:54.332655 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:29:54.332677 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:29:54.334787 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:29:54.343170 (+ 8383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:29:54.343182 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:29:54.343199 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:29:54.343897 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:29:54.344520 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:29:54.344526 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:29:54.344530 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:29:54.344770 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:29:54.345962 (+ 1192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:29:54.345971 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:29:54.346385 (+ 414us) client_negotiation.cc:769] Sending connection context
0927 21:29:54.346695 (+ 310us) client_negotiation.cc:240] Negotiation successful
0927 21:29:54.347295 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18715,"thread_start_us":21346,"threads_started":1}
I20260927 21:29:54.351346 13496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:54.302808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50373 (local address 127.0.97.62:45961)
0927 21:29:54.308850 (+ 6042us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:54.308876 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:54.322269 (+ 13393us) server_negotiation.cc:407] Connection header received
0927 21:29:54.324773 (+ 2504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:54.324792 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:54.325297 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:54.325722 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:29:54.327545 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:54.328896 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:54.330617 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:54.331391 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:54.331605 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:54.335095 (+ 3490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:29:54.335104 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:29:54.335119 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:29:54.335159 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:29:54.342937 (+ 7778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:54.344114 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:54.344125 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:54.344134 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:54.344365 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:54.344906 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:54.344912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:54.344915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:54.345667 (+ 752us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:29:54.345872 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:54.349581 (+ 3709us) server_negotiation.cc:299] Negotiation successful
0927 21:29:54.350050 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5688,"thread_start_us":6057,"threads_started":1}
I20260927 21:29:54.355018 13493 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45961
I20260927 21:29:54.355352 13493 heartbeater.cc:464] Registering TS with master...
I20260927 21:29:54.356235 13493 heartbeater.cc:511] Master 127.0.97.62:45961 requested a full tablet report, sending...
W20260927 21:29:54.359781 13493 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45961 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 21:29:54.360044 13493 heartbeater.cc:464] Registering TS with master...
I20260927 21:29:54.360355 13493 heartbeater.cc:511] Master 127.0.97.62:45961 requested a full tablet report, sending...
I20260927 21:29:54.361783 13493 heartbeater.cc:464] Registering TS with master...
I20260927 21:29:54.362085 13493 heartbeater.cc:511] Master 127.0.97.62:45961 requested a full tablet report, sending...
W20260927 21:29:54.363329 13493 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:29:54.727795 13498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:29:54.728226 13498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:29:54.728384 13498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:29:54.736913 13498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:29:54.737031 13498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:29:54.737272 13498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:29:54.748768 13498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45085/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36273
--tserver_master_addrs=127.0.97.62:45961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:29:54.750660 13498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:29:54.752396 13498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:29:54.768177 13510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:29:54.768216 13511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:29:54.769048 13513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:29:54.770306 13498 server_base.cc:1034] running on GCE node
I20260927 21:29:54.771090 13498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:29:54.772385 13498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:29:54.773679 13498 hybrid_clock.cc:648] HybridClock initialized: now 1790544594773572 us; error 62 us; skew 500 ppm
I20260927 21:29:54.774051 13498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:29:54 dist-test-slave-bxm0 krb5kdc[12285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:29:54.778133 13498 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 21:29:54.783563 13498 webserver.cc:460] Webserver started at http://127.0.97.2:36273/ using document root <none> and password file <none>
I20260927 21:29:54.784354 13498 fs_manager.cc:359] Metadata directory not provided
I20260927 21:29:54.784523 13498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 21:29:54 dist-test-slave-bxm0 krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 21:29:54.812713 13498 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 21:29:54.815810 13523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:29:54.817310 13498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:29:54.817507 13498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "1134e53ebae94f2bafdd4709c20c4b27"
format_stamp: "Formatted at 2026-09-27 21:29:51 on dist-test-slave-bxm0"
server_key: "11483d4df982401cb14a50ea6cf66797"
server_key_iv: "d29781be021a1ab2d04e4de6c5beca32"
server_key_version: "kuduclusterkey@0"
I20260927 21:29:54.818569 13498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:29:54.835785 13498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:29:54.840409 13498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:29:54.840777 13498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:29:54.842152 13498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:29:54.843807 13498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:29:54.843937 13498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:29:54.844076 13498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:29:54.844174 13498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:29:54.963722 13498 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33023
I20260927 21:29:54.963785 13637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33023 every 8 connection(s)
I20260927 21:29:54.965747 13498 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:29:54.972440 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 13498
I20260927 21:29:54.973239 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45085/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36505
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45961
--webserver_port=43269
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45085/kms with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:29:54 dist-test-slave-bxm0 krb5kdc[12285](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790544594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 21:29:54 dist-test-slave-bxm0 krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:29:55.013592 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:54.969474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45961 (local address 127.0.97.2:50537)
0927 21:29:54.982083 (+ 12609us) negotiation.cc:107] Waiting for socket to connect
0927 21:29:54.982144 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 21:29:54.985035 (+ 2891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:29:54.987493 (+ 2458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:29:54.987508 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:29:54.988817 (+ 1309us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:29:54.989727 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:54.989755 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:54.991400 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:54.991405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:54.993461 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:54.993486 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:54.994802 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:54.994810 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:54.995108 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:54.995926 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:29:54.995957 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:29:54.998617 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:29:55.008455 (+ 9838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:29:55.008467 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:29:55.008507 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:29:55.009216 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:29:55.009750 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:29:55.009755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:29:55.009758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:29:55.010027 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:29:55.010774 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:29:55.010782 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:29:55.011245 (+ 463us) client_negotiation.cc:769] Sending connection context
0927 21:29:55.011553 (+ 308us) client_negotiation.cc:240] Negotiation successful
0927 21:29:55.012160 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12251,"thread_start_us":9198,"threads_started":1}
I20260927 21:29:55.014737 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:54.968900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50537 (local address 127.0.97.62:45961)
0927 21:29:54.986582 (+ 17682us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:54.986589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:54.986617 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:29:54.986773 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:54.986778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:54.986876 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:54.987012 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:29:54.989967 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:54.991190 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:54.993708 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:54.994376 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:54.994512 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:54.998920 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:29:54.998928 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:29:54.998935 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:29:54.998976 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:29:55.008210 (+ 9234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:55.009393 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:55.009399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:55.009403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:55.009580 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:55.010165 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:55.010171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:55.010174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:55.010532 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:29:55.010679 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:55.014126 (+ 3447us) server_negotiation.cc:299] Negotiation successful
0927 21:29:55.014378 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17539,"thread_start_us":17408,"threads_started":1}
I20260927 21:29:55.024547 13638 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45961
I20260927 21:29:55.024912 13638 heartbeater.cc:464] Registering TS with master...
I20260927 21:29:55.025876 13638 heartbeater.cc:511] Master 127.0.97.62:45961 requested a full tablet report, sending...
I20260927 21:29:55.028151 13296 ts_manager.cc:194] Registered new tserver with Master: 1134e53ebae94f2bafdd4709c20c4b27 (127.0.97.2:33023)
I20260927 21:29:55.030440 13296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50537
I20260927 21:29:55.363493 13493 heartbeater.cc:464] Registering TS with master...
I20260927 21:29:55.363914 13493 heartbeater.cc:511] Master 127.0.97.62:45961 requested a full tablet report, sending...
I20260927 21:29:55.365216 13296 ts_manager.cc:194] Registered new tserver with Master: 8dafde60964d43b2b14b4ff7a5a9fb37 (127.0.97.1:41611)
I20260927 21:29:55.366443 13296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50373
W20260927 21:29:55.390643 13642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:29:55.391039 13642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:29:55.391237 13642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:29:55.402506 13642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:29:55.402657 13642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:29:55.402892 13642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:29:55.418188 13642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45085/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43269
--tserver_master_addrs=127.0.97.62:45961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:29:55.420069 13642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:29:55.421821 13642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 27 21:29:55 dist-test-slave-bxm0 krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260927 21:29:55.442519 13648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:29:55.450984 13649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:29:55.451982 13651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:29:55.452644 13642 server_base.cc:1034] running on GCE node
I20260927 21:29:55.453471 13642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:29:55.457068 13642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:29:55.458341 13642 hybrid_clock.cc:648] HybridClock initialized: now 1790544595458274 us; error 64 us; skew 500 ppm
I20260927 21:29:55.458771 13642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:29:55 dist-test-slave-bxm0 krb5kdc[12285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:29:55.462944 13642 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 21:29:55.477151 13642 webserver.cc:460] Webserver started at http://127.0.97.3:43269/ using document root <none> and password file <none>
I20260927 21:29:55.478363 13642 fs_manager.cc:359] Metadata directory not provided
I20260927 21:29:55.478607 13642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 21:29:55 dist-test-slave-bxm0 krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 21:29:55.525353 13642 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260927 21:29:55.535163 13659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:29:55.537086 13642 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.007s
I20260927 21:29:55.537338 13642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "0ebceed10feb4d1da01bfcd9a7ad1b1a"
format_stamp: "Formatted at 2026-09-27 21:29:52 on dist-test-slave-bxm0"
server_key: "8d81ce058256998461d914404c97d421"
server_key_iv: "334206cd928b81e53ab52386d7d3f082"
server_key_version: "kuduclusterkey@0"
I20260927 21:29:55.538844 13642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:29:55.575294 13642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:29:55.580554 13642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:29:55.580914 13642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:29:55.609016 13642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:29:55.617604 13642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:29:55.617864 13642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 21:29:55.618046 13642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:29:55.618173 13642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:29:55 dist-test-slave-bxm0 krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 21:29:56.035235 13638 heartbeater.cc:503] Master 127.0.97.62:45961 was elected leader, sending a full tablet report...
I20260927 21:29:56.278643 13642 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36505
I20260927 21:29:56.278710 13779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36505 every 8 connection(s)
I20260927 21:29:56.280583 13642 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:29:56.285197 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 13642
Sep 27 21:29:56 dist-test-slave-bxm0 krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:29:56.316831 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:56.286613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36046 (local address 127.0.97.62:45961)
0927 21:29:56.294784 (+ 8171us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:56.294789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:56.294810 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:29:56.294881 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:56.294899 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:56.294997 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:56.295154 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:29:56.299278 (+ 4124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.300374 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.302707 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.303401 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.303527 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:56.307095 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:29:56.307100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:29:56.307105 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:29:56.307136 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:29:56.312454 (+ 5318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:56.313491 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:56.313496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:56.313500 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:56.313990 (+ 490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:56.314441 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:56.314446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:56.314448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:56.314782 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:29:56.315014 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:56.316205 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0927 21:29:56.316448 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8063,"thread_start_us":7529,"threads_started":1}
I20260927 21:29:56.321923 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:56.284619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44905 (local address 127.0.97.62:45961)
0927 21:29:56.294420 (+ 9801us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:56.294427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:56.298757 (+ 4330us) server_negotiation.cc:407] Connection header received
0927 21:29:56.298823 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:56.298828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:56.298934 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:56.299041 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:29:56.302633 (+ 3592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.303901 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.306140 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.306727 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.306841 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:56.313598 (+ 6757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:29:56.313645 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:29:56.313650 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:29:56.313707 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:29:56.317651 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:56.319022 (+ 1371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:56.319027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:56.319031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:56.319182 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:56.319918 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:56.319922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:56.319926 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:56.320254 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:29:56.320416 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:56.321292 (+ 876us) server_negotiation.cc:299] Negotiation successful
0927 21:29:56.321524 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9638,"thread_start_us":9927,"threads_started":1}
I20260927 21:29:56.323100 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:56.284084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45961 (local address 127.0.97.3:44905)
0927 21:29:56.295870 (+ 11786us) negotiation.cc:107] Waiting for socket to connect
0927 21:29:56.295949 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0927 21:29:56.298269 (+ 2320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:29:56.300726 (+ 2457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:29:56.300739 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:29:56.301413 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:29:56.302407 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:56.302432 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.304083 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.304088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:56.305942 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:56.305970 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.307991 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.307998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:56.308289 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:56.309601 (+ 1312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:29:56.309658 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:29:56.313271 (+ 3613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:29:56.317992 (+ 4721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:29:56.318002 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:29:56.318023 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:29:56.318881 (+ 858us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:29:56.319323 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:29:56.319328 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:29:56.319331 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:29:56.319785 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:29:56.320416 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:29:56.320425 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:29:56.320847 (+ 422us) client_negotiation.cc:769] Sending connection context
0927 21:29:56.321151 (+ 304us) client_negotiation.cc:240] Negotiation successful
0927 21:29:56.321703 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11464,"thread_start_us":11248,"threads_started":1}
I20260927 21:29:56.324523 13780 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45961
I20260927 21:29:56.324847 13780 heartbeater.cc:464] Registering TS with master...
I20260927 21:29:56.325767 13780 heartbeater.cc:511] Master 127.0.97.62:45961 requested a full tablet report, sending...
I20260927 21:29:56.327443 13296 ts_manager.cc:194] Registered new tserver with Master: 0ebceed10feb4d1da01bfcd9a7ad1b1a (127.0.97.3:36505)
I20260927 21:29:56.328666 13296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44905
I20260927 21:29:56.332212 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:29:56.352737 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:56.335796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36050 (local address 127.0.97.62:45961)
0927 21:29:56.335968 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:56.335975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:56.336317 (+ 342us) server_negotiation.cc:407] Connection header received
0927 21:29:56.336486 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:56.336491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:56.336583 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:56.336685 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:29:56.338601 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.339668 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.341519 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.342149 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.342295 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:56.345543 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:29:56.345549 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:29:56.345554 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:29:56.345605 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:29:56.348971 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:56.349782 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:56.349788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:56.349792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:56.349938 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:56.350377 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:56.350382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:56.350386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:56.350749 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:29:56.350997 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:56.352189 (+ 1192us) server_negotiation.cc:299] Negotiation successful
0927 21:29:56.352409 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:29:56.359639 13296 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 21:29:56.370048 13493 heartbeater.cc:503] Master 127.0.97.62:45961 was elected leader, sending a full tablet report...
W20260927 21:29:56.370467 13296 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:29:56.433176 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:56.401460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41611 (local address 127.0.0.1:39908)
0927 21:29:56.406644 (+ 5184us) negotiation.cc:107] Waiting for socket to connect
0927 21:29:56.406672 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 21:29:56.406827 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:29:56.424238 (+ 17411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:29:56.424245 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:29:56.424276 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:29:56.424800 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:56.424821 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.426759 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.426764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:56.429322 (+ 2558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:56.429339 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.431699 (+ 2360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.431705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:56.432137 (+ 432us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:56.432198 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 21:29:56.432342 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 21:29:56.432613 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4995,"mutex_wait_us":126,"thread_start_us":5478,"threads_started":1}
I20260927 21:29:56.434664 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:56.401889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39908 (local address 127.0.97.1:41611)
0927 21:29:56.422477 (+ 20588us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:56.422488 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:56.422525 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:29:56.422631 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:56.422659 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:56.423722 (+ 1063us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:56.423959 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:29:56.425074 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.426583 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.429521 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.431152 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.431289 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:56.431544 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:56.433573 (+ 2029us) server_negotiation.cc:299] Negotiation successful
0927 21:29:56.434297 (+ 724us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20374,"thread_start_us":20191,"threads_started":1}
I20260927 21:29:56.437862 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:56.407405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36505 (local address 127.0.0.1:38154)
0927 21:29:56.422807 (+ 15402us) negotiation.cc:107] Waiting for socket to connect
0927 21:29:56.422836 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 21:29:56.422977 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:29:56.424434 (+ 1457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:29:56.424438 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:29:56.424461 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:29:56.424852 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:56.424863 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.428221 (+ 3358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.428227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:56.430249 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:56.430266 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.437049 (+ 6783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.437058 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:56.437241 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:56.437274 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 21:29:56.437367 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 21:29:56.437499 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15275,"thread_start_us":15396,"threads_started":1}
I20260927 21:29:56.438714 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:56.407789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.0.97.3:36505)
0927 21:29:56.423559 (+ 15770us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:56.423567 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:56.423599 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:29:56.423694 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:56.423701 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:56.423946 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:56.424125 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:29:56.426447 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.428041 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.435063 (+ 7022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.436499 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.436632 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:56.436887 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:56.438208 (+ 1321us) server_negotiation.cc:299] Negotiation successful
0927 21:29:56.438383 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15598,"thread_start_us":15436,"threads_started":1}
I20260927 21:29:56.439257 13428 tablet_service.cc:1467] Processing CreateTablet for tablet 7be911a423ef4a34a64776f957d854c2 (DEFAULT_TABLE table=test-table [id=a89f8f7b7b4a4e84858447c7a75405b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:29:56.441044 13428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be911a423ef4a34a64776f957d854c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:29:56.441594 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:56.386979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33023 (local address 127.0.0.1:50826)
0927 21:29:56.400080 (+ 13101us) negotiation.cc:107] Waiting for socket to connect
0927 21:29:56.400139 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 21:29:56.400350 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:29:56.419476 (+ 19126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:29:56.419490 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:29:56.419558 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:29:56.420162 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:56.420181 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.430462 (+ 10281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.430467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:56.432491 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:56.432503 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.434524 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.434533 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:56.434729 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:56.434772 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 21:29:56.434840 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 21:29:56.441228 (+ 6388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12810,"mutex_wait_us":21,"thread_start_us":13565,"threads_started":1}
I20260927 21:29:56.443090 13708 tablet_service.cc:1467] Processing CreateTablet for tablet 7be911a423ef4a34a64776f957d854c2 (DEFAULT_TABLE table=test-table [id=a89f8f7b7b4a4e84858447c7a75405b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:29:56.444765 13708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be911a423ef4a34a64776f957d854c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:29:56.451359 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:56.387267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50826 (local address 127.0.97.2:33023)
0927 21:29:56.404210 (+ 16943us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:56.404226 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:56.404267 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:29:56.404418 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:56.404430 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:56.404799 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:56.420668 (+ 15869us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:29:56.421038 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.426242 (+ 5204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.432699 (+ 6457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.434272 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.450281 (+ 16009us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:56.450583 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:56.450670 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 21:29:56.450865 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16703,"thread_start_us":16495,"threads_started":1}
I20260927 21:29:56.455402 13573 tablet_service.cc:1467] Processing CreateTablet for tablet 7be911a423ef4a34a64776f957d854c2 (DEFAULT_TABLE table=test-table [id=a89f8f7b7b4a4e84858447c7a75405b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:29:56.457070 13573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be911a423ef4a34a64776f957d854c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:29:56.466816 13801 tablet_bootstrap.cc:492] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37: Bootstrap starting.
I20260927 21:29:56.472983 13801 tablet_bootstrap.cc:654] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37: Neither blocks nor log segments found. Creating new log.
I20260927 21:29:56.474778 13801 log.cc:824] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37: Log is configured to *not* fsync() on all Append() calls
I20260927 21:29:56.484666 13801 tablet_bootstrap.cc:492] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37: No bootstrap required, opened a new log
I20260927 21:29:56.485026 13801 ts_tablet_manager.cc:1397] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.004s
I20260927 21:29:56.490134 13804 tablet_bootstrap.cc:492] T 7be911a423ef4a34a64776f957d854c2 P 1134e53ebae94f2bafdd4709c20c4b27: Bootstrap starting.
I20260927 21:29:56.493014 13803 tablet_bootstrap.cc:492] T 7be911a423ef4a34a64776f957d854c2 P 0ebceed10feb4d1da01bfcd9a7ad1b1a: Bootstrap starting.
I20260927 21:29:56.497319 13801 raft_consensus.cc:348] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1134e53ebae94f2bafdd4709c20c4b27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } } peers { permanent_uuid: "8dafde60964d43b2b14b4ff7a5a9fb37" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41611 } } peers { permanent_uuid: "0ebceed10feb4d1da01bfcd9a7ad1b1a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36505 } }
I20260927 21:29:56.497931 13801 raft_consensus.cc:374] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:29:56.498030 13801 raft_consensus.cc:729] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dafde60964d43b2b14b4ff7a5a9fb37, State: Initialized, Role: FOLLOWER
I20260927 21:29:56.498870 13801 consensus_queue.cc:260] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1134e53ebae94f2bafdd4709c20c4b27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } } peers { permanent_uuid: "8dafde60964d43b2b14b4ff7a5a9fb37" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41611 } } peers { permanent_uuid: "0ebceed10feb4d1da01bfcd9a7ad1b1a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36505 } }
I20260927 21:29:56.503100 13803 tablet_bootstrap.cc:654] T 7be911a423ef4a34a64776f957d854c2 P 0ebceed10feb4d1da01bfcd9a7ad1b1a: Neither blocks nor log segments found. Creating new log.
I20260927 21:29:56.504791 13804 tablet_bootstrap.cc:654] T 7be911a423ef4a34a64776f957d854c2 P 1134e53ebae94f2bafdd4709c20c4b27: Neither blocks nor log segments found. Creating new log.
I20260927 21:29:56.506624 13804 log.cc:824] T 7be911a423ef4a34a64776f957d854c2 P 1134e53ebae94f2bafdd4709c20c4b27: Log is configured to *not* fsync() on all Append() calls
I20260927 21:29:56.507257 13801 ts_tablet_manager.cc:1428] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37: Time spent starting tablet: real 0.022s user 0.018s sys 0.004s
I20260927 21:29:56.508693 13803 log.cc:824] T 7be911a423ef4a34a64776f957d854c2 P 0ebceed10feb4d1da01bfcd9a7ad1b1a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:29:56.519655 13804 tablet_bootstrap.cc:492] T 7be911a423ef4a34a64776f957d854c2 P 1134e53ebae94f2bafdd4709c20c4b27: No bootstrap required, opened a new log
I20260927 21:29:56.520114 13804 ts_tablet_manager.cc:1397] T 7be911a423ef4a34a64776f957d854c2 P 1134e53ebae94f2bafdd4709c20c4b27: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.000s
I20260927 21:29:56.520551 13803 tablet_bootstrap.cc:492] T 7be911a423ef4a34a64776f957d854c2 P 0ebceed10feb4d1da01bfcd9a7ad1b1a: No bootstrap required, opened a new log
I20260927 21:29:56.520828 13803 ts_tablet_manager.cc:1397] T 7be911a423ef4a34a64776f957d854c2 P 0ebceed10feb4d1da01bfcd9a7ad1b1a: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.004s
I20260927 21:29:56.531888 13803 raft_consensus.cc:348] T 7be911a423ef4a34a64776f957d854c2 P 0ebceed10feb4d1da01bfcd9a7ad1b1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1134e53ebae94f2bafdd4709c20c4b27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } } peers { permanent_uuid: "8dafde60964d43b2b14b4ff7a5a9fb37" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41611 } } peers { permanent_uuid: "0ebceed10feb4d1da01bfcd9a7ad1b1a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36505 } }
I20260927 21:29:56.531893 13804 raft_consensus.cc:348] T 7be911a423ef4a34a64776f957d854c2 P 1134e53ebae94f2bafdd4709c20c4b27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1134e53ebae94f2bafdd4709c20c4b27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } } peers { permanent_uuid: "8dafde60964d43b2b14b4ff7a5a9fb37" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41611 } } peers { permanent_uuid: "0ebceed10feb4d1da01bfcd9a7ad1b1a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36505 } }
I20260927 21:29:56.532464 13803 raft_consensus.cc:374] T 7be911a423ef4a34a64776f957d854c2 P 0ebceed10feb4d1da01bfcd9a7ad1b1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:29:56.532562 13803 raft_consensus.cc:729] T 7be911a423ef4a34a64776f957d854c2 P 0ebceed10feb4d1da01bfcd9a7ad1b1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ebceed10feb4d1da01bfcd9a7ad1b1a, State: Initialized, Role: FOLLOWER
I20260927 21:29:56.532967 13804 raft_consensus.cc:374] T 7be911a423ef4a34a64776f957d854c2 P 1134e53ebae94f2bafdd4709c20c4b27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:29:56.533181 13804 raft_consensus.cc:729] T 7be911a423ef4a34a64776f957d854c2 P 1134e53ebae94f2bafdd4709c20c4b27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1134e53ebae94f2bafdd4709c20c4b27, State: Initialized, Role: FOLLOWER
I20260927 21:29:56.533264 13803 consensus_queue.cc:260] T 7be911a423ef4a34a64776f957d854c2 P 0ebceed10feb4d1da01bfcd9a7ad1b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1134e53ebae94f2bafdd4709c20c4b27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } } peers { permanent_uuid: "8dafde60964d43b2b14b4ff7a5a9fb37" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41611 } } peers { permanent_uuid: "0ebceed10feb4d1da01bfcd9a7ad1b1a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36505 } }
I20260927 21:29:56.534241 13804 consensus_queue.cc:260] T 7be911a423ef4a34a64776f957d854c2 P 1134e53ebae94f2bafdd4709c20c4b27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1134e53ebae94f2bafdd4709c20c4b27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } } peers { permanent_uuid: "8dafde60964d43b2b14b4ff7a5a9fb37" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41611 } } peers { permanent_uuid: "0ebceed10feb4d1da01bfcd9a7ad1b1a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36505 } }
I20260927 21:29:56.540110 13804 ts_tablet_manager.cc:1428] T 7be911a423ef4a34a64776f957d854c2 P 1134e53ebae94f2bafdd4709c20c4b27: Time spent starting tablet: real 0.020s user 0.010s sys 0.007s
I20260927 21:29:56.548238 13780 heartbeater.cc:503] Master 127.0.97.62:45961 was elected leader, sending a full tablet report...
I20260927 21:29:56.548941 13805 raft_consensus.cc:482] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:29:56.549208 13805 raft_consensus.cc:504] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1134e53ebae94f2bafdd4709c20c4b27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } } peers { permanent_uuid: "8dafde60964d43b2b14b4ff7a5a9fb37" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41611 } } peers { permanent_uuid: "0ebceed10feb4d1da01bfcd9a7ad1b1a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36505 } }
I20260927 21:29:56.551417 13805 leader_election.cc:288] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1134e53ebae94f2bafdd4709c20c4b27 (127.0.97.2:33023), 0ebceed10feb4d1da01bfcd9a7ad1b1a (127.0.97.3:36505)
I20260927 21:29:56.552122 13803 ts_tablet_manager.cc:1428] T 7be911a423ef4a34a64776f957d854c2 P 0ebceed10feb4d1da01bfcd9a7ad1b1a: Time spent starting tablet: real 0.031s user 0.016s sys 0.014s
W20260927 21:29:56.552541 13494 tablet.cc:2369] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:29:56.563962 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:56.552366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48423 (local address 127.0.97.2:33023)
0927 21:29:56.553527 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:56.553537 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:56.555490 (+ 1953us) server_negotiation.cc:407] Connection header received
0927 21:29:56.555790 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:56.555796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:56.555888 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:56.555952 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:29:56.556770 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.558101 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.561574 (+ 3473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.562754 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.562888 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:56.562991 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:56.563576 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 21:29:56.563708 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":947}
I20260927 21:29:56.564774 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:56.551755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33023 (local address 127.0.97.1:48423)
0927 21:29:56.555384 (+ 3629us) negotiation.cc:107] Waiting for socket to connect
0927 21:29:56.555406 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:29:56.555622 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:29:56.556153 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:29:56.556157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:29:56.556185 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:29:56.556609 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:56.556623 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.558337 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.558346 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:56.561334 (+ 2988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:56.561368 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.563171 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.563178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:56.563377 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:56.563414 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 21:29:56.564255 (+ 841us) client_negotiation.cc:240] Negotiation successful
0927 21:29:56.564418 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3487,"thread_start_us":4404,"threads_started":1}
I20260927 21:29:56.565881 13593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be911a423ef4a34a64776f957d854c2" candidate_uuid: "8dafde60964d43b2b14b4ff7a5a9fb37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1134e53ebae94f2bafdd4709c20c4b27" is_pre_election: true
I20260927 21:29:56.566536 13593 raft_consensus.cc:2454] T 7be911a423ef4a34a64776f957d854c2 P 1134e53ebae94f2bafdd4709c20c4b27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dafde60964d43b2b14b4ff7a5a9fb37 in term 0.
I20260927 21:29:56.567457 13381 leader_election.cc:302] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1134e53ebae94f2bafdd4709c20c4b27, 8dafde60964d43b2b14b4ff7a5a9fb37; no voters:
I20260927 21:29:56.567907 13805 raft_consensus.cc:2781] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:29:56.568080 13805 raft_consensus.cc:482] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:29:56.568210 13805 raft_consensus.cc:3037] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:29:56.568935 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:56.557091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40759 (local address 127.0.97.3:36505)
0927 21:29:56.557258 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:56.557263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:56.562202 (+ 4939us) server_negotiation.cc:407] Connection header received
0927 21:29:56.562393 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:56.562399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:56.562475 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:56.562542 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:29:56.563345 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.564394 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.566703 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.567797 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.567891 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:56.567984 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:56.568568 (+ 584us) server_negotiation.cc:299] Negotiation successful
0927 21:29:56.568699 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 21:29:56.569687 13811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:56.556692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36505 (local address 127.0.97.1:40759)
0927 21:29:56.562069 (+ 5377us) negotiation.cc:107] Waiting for socket to connect
0927 21:29:56.562089 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:29:56.562222 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:29:56.562746 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:29:56.562751 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:29:56.562779 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:29:56.563162 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:56.563174 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.564571 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.564576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:56.566532 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:29:56.566547 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.568187 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.568197 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:29:56.568365 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:56.568412 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 21:29:56.569160 (+ 748us) client_negotiation.cc:240] Negotiation successful
0927 21:29:56.569313 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5258,"thread_start_us":5059,"threads_started":1}
I20260927 21:29:56.571447 13805 raft_consensus.cc:504] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1134e53ebae94f2bafdd4709c20c4b27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } } peers { permanent_uuid: "8dafde60964d43b2b14b4ff7a5a9fb37" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41611 } } peers { permanent_uuid: "0ebceed10feb4d1da01bfcd9a7ad1b1a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36505 } }
I20260927 21:29:56.572203 13805 leader_election.cc:288] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37 [CANDIDATE]: Term 1 election: Requested vote from peers 1134e53ebae94f2bafdd4709c20c4b27 (127.0.97.2:33023), 0ebceed10feb4d1da01bfcd9a7ad1b1a (127.0.97.3:36505)
I20260927 21:29:56.572808 13593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be911a423ef4a34a64776f957d854c2" candidate_uuid: "8dafde60964d43b2b14b4ff7a5a9fb37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1134e53ebae94f2bafdd4709c20c4b27"
I20260927 21:29:56.573028 13593 raft_consensus.cc:3037] T 7be911a423ef4a34a64776f957d854c2 P 1134e53ebae94f2bafdd4709c20c4b27 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:29:56.573493 13728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be911a423ef4a34a64776f957d854c2" candidate_uuid: "8dafde60964d43b2b14b4ff7a5a9fb37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ebceed10feb4d1da01bfcd9a7ad1b1a" is_pre_election: true
I20260927 21:29:56.574122 13728 raft_consensus.cc:2454] T 7be911a423ef4a34a64776f957d854c2 P 0ebceed10feb4d1da01bfcd9a7ad1b1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dafde60964d43b2b14b4ff7a5a9fb37 in term 0.
I20260927 21:29:56.573563 13727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be911a423ef4a34a64776f957d854c2" candidate_uuid: "8dafde60964d43b2b14b4ff7a5a9fb37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ebceed10feb4d1da01bfcd9a7ad1b1a"
I20260927 21:29:56.575879 13727 raft_consensus.cc:3037] T 7be911a423ef4a34a64776f957d854c2 P 0ebceed10feb4d1da01bfcd9a7ad1b1a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:29:56.576262 13593 raft_consensus.cc:2454] T 7be911a423ef4a34a64776f957d854c2 P 1134e53ebae94f2bafdd4709c20c4b27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dafde60964d43b2b14b4ff7a5a9fb37 in term 1.
I20260927 21:29:56.576931 13381 leader_election.cc:302] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1134e53ebae94f2bafdd4709c20c4b27, 8dafde60964d43b2b14b4ff7a5a9fb37; no voters:
I20260927 21:29:56.577598 13805 raft_consensus.cc:2781] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:29:56.579015 13727 raft_consensus.cc:2454] T 7be911a423ef4a34a64776f957d854c2 P 0ebceed10feb4d1da01bfcd9a7ad1b1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dafde60964d43b2b14b4ff7a5a9fb37 in term 1.
I20260927 21:29:56.580055 13805 raft_consensus.cc:686] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37 [term 1 LEADER]: Becoming Leader. State: Replica: 8dafde60964d43b2b14b4ff7a5a9fb37, State: Running, Role: LEADER
I20260927 21:29:56.580663 13805 consensus_queue.cc:237] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1134e53ebae94f2bafdd4709c20c4b27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } } peers { permanent_uuid: "8dafde60964d43b2b14b4ff7a5a9fb37" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41611 } } peers { permanent_uuid: "0ebceed10feb4d1da01bfcd9a7ad1b1a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36505 } }
I20260927 21:29:56.585687 13296 catalog_manager.cc:5495] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dafde60964d43b2b14b4ff7a5a9fb37 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "8dafde60964d43b2b14b4ff7a5a9fb37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1134e53ebae94f2bafdd4709c20c4b27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dafde60964d43b2b14b4ff7a5a9fb37" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebceed10feb4d1da01bfcd9a7ad1b1a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36505 } health_report { overall_health: UNKNOWN } } }
I20260927 21:29:56.638255 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:56.632210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39916 (local address 127.0.97.1:41611)
0927 21:29:56.632403 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:56.632410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:56.632531 (+ 121us) server_negotiation.cc:407] Connection header received
0927 21:29:56.632737 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:56.632743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:56.632853 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:56.633005 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:29:56.633354 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.634442 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.635778 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.636387 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.636513 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:56.636983 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:29:56.637699 (+ 716us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:29:56.637836 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:56.637904 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 21:29:56.637986 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:29:56.647807 13727 raft_consensus.cc:1261] T 7be911a423ef4a34a64776f957d854c2 P 0ebceed10feb4d1da01bfcd9a7ad1b1a [term 1 FOLLOWER]: Refusing update from remote peer 8dafde60964d43b2b14b4ff7a5a9fb37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:29:56.647946 13593 raft_consensus.cc:1261] T 7be911a423ef4a34a64776f957d854c2 P 1134e53ebae94f2bafdd4709c20c4b27 [term 1 FOLLOWER]: Refusing update from remote peer 8dafde60964d43b2b14b4ff7a5a9fb37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:29:56.648872 13805 consensus_queue.cc:1035] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1134e53ebae94f2bafdd4709c20c4b27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:29:56.649546 13812 consensus_queue.cc:1035] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebceed10feb4d1da01bfcd9a7ad1b1a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:29:56.662614 13816 mvcc.cc:205] Tried to move back new op lower bound from 7334070667858350080 to 7334070667598909440. Current Snapshot: MvccSnapshot[applied={T|T < 7334070667858350080}]
I20260927 21:29:56.684237 13296 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:29:56.684536 13296 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:29:56.688777 13296 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ae8cda3649b4706829f4fbf77708129: Sending DeleteTablet for 3 replicas of tablet 7be911a423ef4a34a64776f957d854c2
I20260927 21:29:56.690248 13573 tablet_service.cc:1514] Processing DeleteTablet for tablet 7be911a423ef4a34a64776f957d854c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:29:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50826
I20260927 21:29:56.690640 13428 tablet_service.cc:1514] Processing DeleteTablet for tablet 7be911a423ef4a34a64776f957d854c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:29:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39908
I20260927 21:29:56.691093 13708 tablet_service.cc:1514] Processing DeleteTablet for tablet 7be911a423ef4a34a64776f957d854c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:29:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38154
I20260927 21:29:56.693225 13827 tablet_replica.cc:330] T 7be911a423ef4a34a64776f957d854c2 P 0ebceed10feb4d1da01bfcd9a7ad1b1a: stopping tablet replica
I20260927 21:29:56.693817 13827 raft_consensus.cc:2229] T 7be911a423ef4a34a64776f957d854c2 P 0ebceed10feb4d1da01bfcd9a7ad1b1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:29:56.694415 13827 raft_consensus.cc:2258] T 7be911a423ef4a34a64776f957d854c2 P 0ebceed10feb4d1da01bfcd9a7ad1b1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:29:56.694545 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 13333
I20260927 21:29:56.698840 13827 ts_tablet_manager.cc:1905] T 7be911a423ef4a34a64776f957d854c2 P 0ebceed10feb4d1da01bfcd9a7ad1b1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:29:56.704439 13825 tablet_replica.cc:330] T 7be911a423ef4a34a64776f957d854c2 P 1134e53ebae94f2bafdd4709c20c4b27: stopping tablet replica
I20260927 21:29:56.708387 13825 raft_consensus.cc:2229] T 7be911a423ef4a34a64776f957d854c2 P 1134e53ebae94f2bafdd4709c20c4b27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:29:56.709280 13825 raft_consensus.cc:2258] T 7be911a423ef4a34a64776f957d854c2 P 1134e53ebae94f2bafdd4709c20c4b27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:29:56.710246 13826 tablet_replica.cc:330] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37: stopping tablet replica
I20260927 21:29:56.710724 13826 raft_consensus.cc:2229] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:29:56.712738 13827 ts_tablet_manager.cc:1918] T 7be911a423ef4a34a64776f957d854c2 P 0ebceed10feb4d1da01bfcd9a7ad1b1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:29:56.712922 13827 log.cc:1196] T 7be911a423ef4a34a64776f957d854c2 P 0ebceed10feb4d1da01bfcd9a7ad1b1a: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-2/wal/wals/7be911a423ef4a34a64776f957d854c2
I20260927 21:29:56.713514 13827 ts_tablet_manager.cc:1939] T 7be911a423ef4a34a64776f957d854c2 P 0ebceed10feb4d1da01bfcd9a7ad1b1a: Deleting consensus metadata
I20260927 21:29:56.714944 13825 ts_tablet_manager.cc:1905] T 7be911a423ef4a34a64776f957d854c2 P 1134e53ebae94f2bafdd4709c20c4b27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:29:56.715458 13280 catalog_manager.cc:4841] TS 0ebceed10feb4d1da01bfcd9a7ad1b1a (127.0.97.3:36505): tablet 7be911a423ef4a34a64776f957d854c2 (table test-table [id=a89f8f7b7b4a4e84858447c7a75405b6]) successfully deleted
I20260927 21:29:56.716508 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:56.695870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39922 (local address 127.0.97.1:41611)
0927 21:29:56.696037 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:56.696043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:56.696231 (+ 188us) server_negotiation.cc:407] Connection header received
0927 21:29:56.696431 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:56.696437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:56.696519 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:56.696606 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:29:56.699607 (+ 3001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.700983 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.703804 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.704583 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.704713 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:56.708099 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:29:56.708114 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:29:56.708133 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:29:56.708182 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:29:56.712806 (+ 4624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:56.713963 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:56.713972 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:56.713975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:56.714151 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:56.714581 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:56.714588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:56.714594 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:56.715155 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:29:56.715495 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:56.715844 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 21:29:56.716098 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:29:56.717213 13488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:29:56.723454 13825 ts_tablet_manager.cc:1918] T 7be911a423ef4a34a64776f957d854c2 P 1134e53ebae94f2bafdd4709c20c4b27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:29:56.723639 13825 log.cc:1196] T 7be911a423ef4a34a64776f957d854c2 P 1134e53ebae94f2bafdd4709c20c4b27: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790544390748963-388-0/minicluster-data/ts-1/wal/wals/7be911a423ef4a34a64776f957d854c2
I20260927 21:29:56.724116 13825 ts_tablet_manager.cc:1939] T 7be911a423ef4a34a64776f957d854c2 P 1134e53ebae94f2bafdd4709c20c4b27: Deleting consensus metadata
I20260927 21:29:56.725731 13280 catalog_manager.cc:4841] TS 1134e53ebae94f2bafdd4709c20c4b27 (127.0.97.2:33023): tablet 7be911a423ef4a34a64776f957d854c2 (table test-table [id=a89f8f7b7b4a4e84858447c7a75405b6]) successfully deleted
I20260927 21:29:56.711514 13826 raft_consensus.cc:2258] T 7be911a423ef4a34a64776f957d854c2 P 8dafde60964d43b2b14b4ff7a5a9fb37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:29:56.815652 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 13333
I20260927 21:29:56.833230 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 13498
W20260927 21:29:56.833292 13280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:41611 (error 108)
I20260927 21:29:56.835042 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:56.834660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41611 (local address 127.0.0.1:39938)
0927 21:29:56.834817 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 21:29:56.834909 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 21:29:56.835304 13280 catalog_manager.cc:4584] TS 8dafde60964d43b2b14b4ff7a5a9fb37 (127.0.97.1:41611): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7be911a423ef4a34a64776f957d854c2: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41611: connect: Connection refused (error 111)
I20260927 21:29:56.847414 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:56.834079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50842 (local address 127.0.97.2:33023)
0927 21:29:56.834341 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:56.834349 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:56.834424 (+ 75us) server_negotiation.cc:407] Connection header received
0927 21:29:56.834652 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:56.834656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:56.834720 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:56.834852 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:29:56.836367 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.837079 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.838562 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.838986 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.839126 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:56.841700 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:29:56.841714 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:29:56.841731 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:29:56.841767 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:29:56.844488 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:56.845165 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:56.845173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:56.845175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:56.845359 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:56.845844 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:56.845849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:56.845851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:56.846191 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:29:56.846509 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:56.846820 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 21:29:56.847046 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260927 21:29:56.848097 13633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:29:56.889052 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:56.888569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41611 (local address 127.0.0.1:39940)
0927 21:29:56.888790 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 21:29:56.888885 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 21:29:56.890131 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:56.889793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41611 (local address 127.0.0.1:39956)
0927 21:29:56.889955 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 21:29:56.890032 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 21:29:56.937845 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 13498
I20260927 21:29:56.950824 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:56.950223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41611 (local address 127.0.0.1:39958)
0927 21:29:56.950567 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0927 21:29:56.950660 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260927 21:29:56.951982 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:56.951630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41611 (local address 127.0.0.1:39966)
0927 21:29:56.951782 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 21:29:56.951865 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 21:29:56.955065 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 13642
I20260927 21:29:56.968648 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:56.955808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38158 (local address 127.0.97.3:36505)
0927 21:29:56.955982 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 21:29:56.955987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:29:56.956005 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:29:56.956130 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:29:56.956134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:29:56.956196 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:29:56.956276 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:29:56.957555 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.958456 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.959682 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:29:56.960079 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:29:56.960171 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:29:56.962768 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:29:56.962781 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:29:56.962798 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:29:56.962838 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:29:56.965892 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:56.966646 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:56.966653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:56.966656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:56.966792 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:29:56.967124 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:29:56.967127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:29:56.967129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:29:56.967581 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:29:56.967808 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 21:29:56.968059 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 21:29:56.968274 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:29:56.969355 13775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:29:57.033039 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:57.032562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41611 (local address 127.0.0.1:39972)
0927 21:29:57.032769 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 21:29:57.032855 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 21:29:57.034214 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:29:57.033826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41611 (local address 127.0.0.1:39978)
0927 21:29:57.034033 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 21:29:57.034109 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260927 21:29:57.058127 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 13642
I20260927 21:29:57.074826 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 13263
I20260927 21:29:57.076047 13326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:29:57.187335 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 13263
I20260927 21:29:57.210022 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 21:29:57.546832 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 21:29:57.547075 388 mini_ranger.cc:66] Stopping Ranger...
I20260927 21:29:57.893514 388 mini_ranger.cc:68] Stopped Ranger
2026-09-27 21:29:57.893 UTC [12315] LOG: received smart shutdown request
2026-09-27 21:29:57.897 UTC [12315] LOG: background worker "logical replication launcher" (PID 12324) exited with exit code 1
2026-09-27 21:29:57.897 UTC [12319] LOG: shutting down
2026-09-27 21:29:58.072 UTC [12315] LOG: database system is shut down
2026-09-27T21:29:58Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53226 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:29:58 dist-test-slave-bxm0 krb5kdc[13841](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:29:58 dist-test-slave-bxm0 krb5kdc[13841](info): set up 2 sockets
Sep 27 21:29:58 dist-test-slave-bxm0 krb5kdc[13841](info): commencing operation
krb5kdc: starting...
W20260927 21:29:59.306121 388 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:29:59 dist-test-slave-bxm0 krb5kdc[13841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:29:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:29:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:29:59.557229 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41081
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41081,127.0.97.61:33831,127.0.97.60:43763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:29:59.897094 13854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:29:59.897416 13854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:29:59.897593 13854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:29:59.905336 13854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:29:59.905454 13854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:29:59.905499 13854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:29:59.905565 13854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:29:59.905663 13854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:29:59.915895 13854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41081,127.0.97.61:33831,127.0.97.60:43763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:29:59.917184 13854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:29:59.918639 13854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:29:59.930658 13859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:29:59.930729 13860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:29:59.931425 13854 server_base.cc:1034] running on GCE node
W20260927 21:29:59.931422 13862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:29:59.932337 13854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:29:59.934183 13854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:29:59.935472 13854 hybrid_clock.cc:648] HybridClock initialized: now 1790544599935384 us; error 66 us; skew 500 ppm
I20260927 21:29:59.935850 13854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:29:59 dist-test-slave-bxm0 krb5kdc[13841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:29:59.939038 13854 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:29:59.940604 13854 webserver.cc:460] Webserver started at http://127.0.97.62:41655/ using document root <none> and password file <none>
I20260927 21:29:59.941326 13854 fs_manager.cc:359] Metadata directory not provided
I20260927 21:29:59.941479 13854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:29:59.941859 13854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:29:59.944870 13854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "860cf650a2aa4fe69323f0fdd41ce16d"
format_stamp: "Formatted at 2026-09-27 21:29:59 on dist-test-slave-bxm0"
server_key: "705088c128d69d6c14d08999e5b44b86"
server_key_iv: "1fd6b62d2b01674e579a71532d849b00"
server_key_version: "encryptionkey@0"
I20260927 21:29:59.945739 13854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "860cf650a2aa4fe69323f0fdd41ce16d"
format_stamp: "Formatted at 2026-09-27 21:29:59 on dist-test-slave-bxm0"
server_key: "705088c128d69d6c14d08999e5b44b86"
server_key_iv: "1fd6b62d2b01674e579a71532d849b00"
server_key_version: "encryptionkey@0"
I20260927 21:29:59.951200 13854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 21:29:59.954793 13869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:29:59.956133 13854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:29:59.956327 13854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "860cf650a2aa4fe69323f0fdd41ce16d"
format_stamp: "Formatted at 2026-09-27 21:29:59 on dist-test-slave-bxm0"
server_key: "705088c128d69d6c14d08999e5b44b86"
server_key_iv: "1fd6b62d2b01674e579a71532d849b00"
server_key_version: "encryptionkey@0"
I20260927 21:29:59.956542 13854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:29:59.975783 13854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:29:59.980134 13854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:29:59.980433 13854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:29:59.999434 13854 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41081
I20260927 21:29:59.999527 13920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41081 every 8 connection(s)
I20260927 21:30:00.000857 13854 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:30:00.004731 13921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:30:00.010145 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 13854
I20260927 21:30:00.010378 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:30:00.011154 388 external_mini_cluster.cc:1428] Setting key 5a7aa2eb02fcb7463efaa3b3cf9e61ac
I20260927 21:30:00.012198 13921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41081 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:30:00 dist-test-slave-bxm0 krb5kdc[13841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260927 21:30:00.038721 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.016411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41081 (local address 127.0.0.1:47136)
0927 21:30:00.017780 (+ 1369us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.017831 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:00.019777 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:00.020634 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:00.020643 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:00.021473 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:30:00.022027 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:00.022057 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:00.023602 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:00.023610 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:00.025578 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:00.025596 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:00.026590 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:00.026599 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:00.026831 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:00.027581 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:30:00.027616 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:30:00.030531 (+ 2915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:30:00.034085 (+ 3554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:00.034097 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:00.034111 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:00.034575 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:00.035019 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:00.035025 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:00.035028 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:00.035207 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:00.036085 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:30:00.036093 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:30:00.036318 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 21:30:00.036625 (+ 307us) client_negotiation.cc:240] Negotiation successful
0927 21:30:00.037231 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1081,"mutex_wait_us":146,"spinlock_wait_cycles":61696,"thread_start_us":839,"threads_started":1}
I20260927 21:30:00.038748 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.016420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47136 (local address 127.0.97.62:41081)
0927 21:30:00.017368 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:00.017384 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:00.017969 (+ 585us) server_negotiation.cc:407] Connection header received
0927 21:30:00.020143 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:00.020149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:00.020372 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:00.020537 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:00.022247 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:00.023439 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:00.025751 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:00.026330 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:00.026474 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:00.030779 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:00.030787 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:00.030799 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:00.030833 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:00.033948 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:00.034727 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:00.034733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:00.034736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:00.034903 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:00.035322 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:00.035333 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:00.035337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:00.035930 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:00.036139 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:00.036759 (+ 620us) server_negotiation.cc:299] Negotiation successful
0927 21:30:00.037114 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":682,"thread_start_us":446,"threads_started":1}
I20260927 21:30:00.041980 13921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:30:00.043080 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.042694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33831 (local address 127.0.0.1:51514)
0927 21:30:00.042858 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.042965 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 21:30:00.043329 13871 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33831: connect: Connection refused (error 111)
I20260927 21:30:00.044991 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.044552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33831 (local address 127.0.0.1:51518)
0927 21:30:00.044698 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.044807 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 21:30:00.045212 13921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33831: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33831: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260927 21:30:00.065284 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33831
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41081,127.0.97.61:33831,127.0.97.60:43763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:30:00.094647 13921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 } attempt: 1
I20260927 21:30:00.095892 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.095373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33831 (local address 127.0.0.1:51528)
0927 21:30:00.095598 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.095726 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 21:30:00.097004 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.096601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33831 (local address 127.0.0.1:51542)
0927 21:30:00.096754 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.096840 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 21:30:00.097249 13921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33831: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33831: connect: Connection refused (error 111)
I20260927 21:30:00.165508 13921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 } attempt: 2
I20260927 21:30:00.166841 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.166242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33831 (local address 127.0.0.1:51548)
0927 21:30:00.166538 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.166665 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 21:30:00.167953 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.167528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33831 (local address 127.0.0.1:51556)
0927 21:30:00.167698 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.167785 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 21:30:00.168154 13921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33831: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33831: connect: Connection refused (error 111)
I20260927 21:30:00.259460 13921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 } attempt: 3
I20260927 21:30:00.260730 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.260187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33831 (local address 127.0.0.1:51572)
0927 21:30:00.260403 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.260586 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 21:30:00.261850 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.261417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33831 (local address 127.0.0.1:51574)
0927 21:30:00.261569 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.261709 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 21:30:00.262106 13921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33831: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33831: connect: Connection refused (error 111)
W20260927 21:30:00.394057 13928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:30:00.394363 13928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:30:00.394543 13928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:30:00.401858 13928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:30:00.401986 13928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:30:00.402127 13928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:30:00.402228 13928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:30:00.402307 13928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:30:00.405372 13921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 } attempt: 4
I20260927 21:30:00.406575 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.406059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33831 (local address 127.0.0.1:51586)
0927 21:30:00.406363 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.406447 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260927 21:30:00.407575 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.407234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33831 (local address 127.0.0.1:51588)
0927 21:30:00.407388 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.407458 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 21:30:00.407768 13921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33831: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33831: connect: Connection refused (error 111)
I20260927 21:30:00.412994 13928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41081,127.0.97.61:33831,127.0.97.60:43763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:30:00.414705 13928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:30:00.416114 13928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:30:00.427816 13933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:00.427735 13936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:00.427781 13934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:30:00.428629 13928 server_base.cc:1034] running on GCE node
I20260927 21:30:00.429383 13928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:30:00.430838 13928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:30:00.432080 13928 hybrid_clock.cc:648] HybridClock initialized: now 1790544600432029 us; error 43 us; skew 500 ppm
I20260927 21:30:00.432432 13928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:30:00 dist-test-slave-bxm0 krb5kdc[13841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:30:00.435549 13928 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260927 21:30:00.437084 13928 webserver.cc:460] Webserver started at http://127.0.97.61:45413/ using document root <none> and password file <none>
I20260927 21:30:00.437829 13928 fs_manager.cc:359] Metadata directory not provided
I20260927 21:30:00.437970 13928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:30:00.438284 13928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:30:00.441376 13928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-1/data/instance:
uuid: "8fe1e1831c78480093f9d4b7b0dcedc1"
format_stamp: "Formatted at 2026-09-27 21:30:00 on dist-test-slave-bxm0"
server_key: "dfcc6730cebf635f9d3f2f64e787c92b"
server_key_iv: "ec505f128039896b14765b20b5113f24"
server_key_version: "encryptionkey@0"
I20260927 21:30:00.442224 13928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-1/wal/instance:
uuid: "8fe1e1831c78480093f9d4b7b0dcedc1"
format_stamp: "Formatted at 2026-09-27 21:30:00 on dist-test-slave-bxm0"
server_key: "dfcc6730cebf635f9d3f2f64e787c92b"
server_key_iv: "ec505f128039896b14765b20b5113f24"
server_key_version: "encryptionkey@0"
I20260927 21:30:00.447546 13928 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:30:00.450989 13943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:00.452369 13928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260927 21:30:00.452569 13928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-1/wal
uuid: "8fe1e1831c78480093f9d4b7b0dcedc1"
format_stamp: "Formatted at 2026-09-27 21:30:00 on dist-test-slave-bxm0"
server_key: "dfcc6730cebf635f9d3f2f64e787c92b"
server_key_iv: "ec505f128039896b14765b20b5113f24"
server_key_version: "encryptionkey@0"
I20260927 21:30:00.452780 13928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:30:00.476330 13928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:30:00.481035 13928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:30:00.481416 13928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:30:00.502175 13928 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:33831
I20260927 21:30:00.502216 13994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:33831 every 8 connection(s)
I20260927 21:30:00.503579 13928 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-1/data/info.pb
I20260927 21:30:00.507092 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 13928
I20260927 21:30:00.507290 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-1/wal/instance
I20260927 21:30:00.507732 13995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:30:00.507869 388 external_mini_cluster.cc:1428] Setting key f5e64d1ae4954975b715054ecdade301
I20260927 21:30:00.517576 13995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41081 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 27 21:30:00 dist-test-slave-bxm0 krb5kdc[13841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:30:00.539779 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.520477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47144 (local address 127.0.97.62:41081)
0927 21:30:00.520736 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:00.520745 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:00.522071 (+ 1326us) server_negotiation.cc:407] Connection header received
0927 21:30:00.523981 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:00.523990 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:00.524091 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:00.524219 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:00.525963 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:00.527194 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:00.528777 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:00.529503 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:00.529696 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:00.533696 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:00.533702 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:00.533705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:00.533742 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:00.536454 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:00.537234 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:00.537239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:00.537242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:00.537362 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:00.537916 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:00.537924 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:00.537931 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:00.538276 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:00.538444 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:00.539213 (+ 769us) server_negotiation.cc:299] Negotiation successful
0927 21:30:00.539448 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 21:30:00.540791 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.520934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41081 (local address 127.0.0.1:47144)
0927 21:30:00.521854 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.521911 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:00.523661 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:00.524417 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:00.524426 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:00.524971 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:30:00.525743 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:00.525764 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:00.527371 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:00.527375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:00.528610 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:00.528622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:00.529768 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:00.529773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:00.529991 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:00.530715 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:30:00.530738 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:30:00.533398 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:30:00.536614 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:00.536624 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:00.536642 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:00.537094 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:00.537480 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:00.537485 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:00.537489 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:00.537761 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:00.538427 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:30:00.538435 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:30:00.538823 (+ 388us) client_negotiation.cc:769] Sending connection context
0927 21:30:00.539107 (+ 284us) client_negotiation.cc:240] Negotiation successful
0927 21:30:00.539640 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":384,"threads_started":1}
I20260927 21:30:00.542618 13995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:30:00 dist-test-slave-bxm0 krb5kdc[13841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 21:30:00.559578 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.543510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33831 (local address 127.0.0.1:51594)
0927 21:30:00.543722 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.543745 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:00.543890 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:00.544761 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:00.544764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:00.545180 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:30:00.545319 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:00.545328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:00.547496 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:00.547500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:00.548453 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:00.548469 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:00.549840 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:00.549846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:00.550039 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:00.550725 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:30:00.550743 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:30:00.552960 (+ 2217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:30:00.556800 (+ 3840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:00.556806 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:00.556811 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:00.557351 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:00.557875 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:00.557879 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:00.557882 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:00.558021 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:00.558722 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:30:00.558725 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:30:00.558855 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 21:30:00.558978 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 21:30:00.559231 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":1}
I20260927 21:30:00.559876 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.543537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51594 (local address 127.0.97.61:33831)
0927 21:30:00.544226 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:00.544236 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:00.544269 (+ 33us) server_negotiation.cc:407] Connection header received
0927 21:30:00.544327 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:00.544337 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:00.544604 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:00.544759 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:00.545488 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:00.547322 (+ 1834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:00.548626 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:00.549680 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:00.549831 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:00.553168 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:00.553180 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:00.553195 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:00.553225 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:00.556604 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:00.557494 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:00.557518 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:00.557523 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:00.557724 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:00.558128 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:00.558131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:00.558133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:00.558570 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:00.558731 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:00.559176 (+ 445us) server_negotiation.cc:299] Negotiation successful
0927 21:30:00.559372 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":494,"thread_start_us":310,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260927 21:30:00.561682 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:43763
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41081,127.0.97.61:33831,127.0.97.60:43763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:30:00.562706 13995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:30:00.563809 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.563377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43763 (local address 127.0.0.1:47490)
0927 21:30:00.563540 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.563657 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 21:30:00.564153 13945 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
I20260927 21:30:00.565824 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.565420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43763 (local address 127.0.0.1:47502)
0927 21:30:00.565578 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.565715 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 21:30:00.566120 13995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43763: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
I20260927 21:30:00.615468 13995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } attempt: 1
I20260927 21:30:00.616787 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.616319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43763 (local address 127.0.0.1:47512)
0927 21:30:00.616526 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.616633 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 21:30:00.618074 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.617547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43763 (local address 127.0.0.1:47518)
0927 21:30:00.617833 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.617910 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260927 21:30:00.618381 13995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43763: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
I20260927 21:30:00.686672 13995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } attempt: 2
I20260927 21:30:00.688526 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.687650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43763 (local address 127.0.0.1:47534)
0927 21:30:00.687983 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.688250 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 21:30:00.689718 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.689271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43763 (local address 127.0.0.1:47538)
0927 21:30:00.689429 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.689523 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 21:30:00.689947 13995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43763: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
I20260927 21:30:00.707041 13921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 } attempt: 5
Sep 27 21:30:00 dist-test-slave-bxm0 krb5kdc[13841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 21:30:00.721444 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.708154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51596 (local address 127.0.97.61:33831)
0927 21:30:00.708383 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:00.708392 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:00.708414 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:30:00.708477 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:00.708481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:00.708561 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:00.708673 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:00.709543 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:00.710321 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:00.711528 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:00.712014 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:00.712158 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:00.715918 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:00.715923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:00.715926 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:00.715962 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:00.718820 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:00.719509 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:00.719515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:00.719519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:00.719647 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:00.720027 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:00.720033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:00.720036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:00.720349 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:00.720506 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:00.720902 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 21:30:00.721119 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 21:30:00.721462 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.707903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33831 (local address 127.0.0.1:51596)
0927 21:30:00.708109 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.708137 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:00.708288 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:00.708721 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:00.708727 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:00.709161 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:30:00.709372 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:00.709381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:00.710478 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:00.710484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:00.711370 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:00.711382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:00.712174 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:00.712180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:00.712387 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:00.713150 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:30:00.713169 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:30:00.715681 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:30:00.718975 (+ 3294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:00.718981 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:00.718986 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:00.719384 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:00.719765 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:00.719771 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:00.719775 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:00.719917 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:00.720471 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:30:00.720476 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:30:00.720634 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 21:30:00.720787 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 21:30:00.721093 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 21:30:00.723277 13921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:30:00.724390 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.723991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43763 (local address 127.0.0.1:47554)
0927 21:30:00.724163 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.724241 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 21:30:00.725394 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.725065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43763 (local address 127.0.0.1:47570)
0927 21:30:00.725195 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.725273 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 21:30:00.725713 13921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43763: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
I20260927 21:30:00.777024 13921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } attempt: 1
I20260927 21:30:00.778463 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.777931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43763 (local address 127.0.0.1:47572)
0927 21:30:00.778162 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.778281 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 21:30:00.779614 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.779237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43763 (local address 127.0.0.1:47588)
0927 21:30:00.779402 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.779498 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 21:30:00.779862 13921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43763: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
I20260927 21:30:00.781175 13995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } attempt: 3
I20260927 21:30:00.782347 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.781871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43763 (local address 127.0.0.1:47596)
0927 21:30:00.782081 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.782177 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 21:30:00.783396 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.782938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43763 (local address 127.0.0.1:47610)
0927 21:30:00.783174 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.783262 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260927 21:30:00.783624 13995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43763: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
I20260927 21:30:00.848121 13921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } attempt: 2
I20260927 21:30:00.849301 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.848795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43763 (local address 127.0.0.1:47626)
0927 21:30:00.849035 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.849126 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 21:30:00.850531 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.850075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43763 (local address 127.0.0.1:47634)
0927 21:30:00.850247 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.850378 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 21:30:00.850766 13921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43763: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
W20260927 21:30:00.905949 14002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:30:00.906241 14002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:30:00.906432 14002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:30:00.914028 14002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:30:00.914140 14002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:30:00.914217 14002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:30:00.914326 14002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:30:00.914405 14002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:30:00.924606 14002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41081,127.0.97.61:33831,127.0.97.60:43763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:43763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:30:00.926471 14002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:30:00.926831 13995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } attempt: 4
I20260927 21:30:00.928040 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.927461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43763 (local address 127.0.0.1:47638)
0927 21:30:00.927711 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.927831 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 21:30:00.928162 14002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:30:00.929050 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.928681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43763 (local address 127.0.0.1:47642)
0927 21:30:00.928843 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.928924 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 21:30:00.929267 13995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43763: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
W20260927 21:30:00.940550 14007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:00.940551 14008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:00.940616 14010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:30:00.941250 14002 server_base.cc:1034] running on GCE node
I20260927 21:30:00.942227 14002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:30:00.943696 14002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:30:00.944928 14002 hybrid_clock.cc:648] HybridClock initialized: now 1790544600944867 us; error 59 us; skew 500 ppm
I20260927 21:30:00.945293 14002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:30:00 dist-test-slave-bxm0 krb5kdc[13841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:30:00.948482 14002 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260927 21:30:00.950057 14002 webserver.cc:460] Webserver started at http://127.0.97.60:38941/ using document root <none> and password file <none>
I20260927 21:30:00.950783 14002 fs_manager.cc:359] Metadata directory not provided
I20260927 21:30:00.950933 14002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:30:00.951248 14002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:30:00.954607 14002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-2/data/instance:
uuid: "2003c5fc8ddf40498343730960161195"
format_stamp: "Formatted at 2026-09-27 21:30:00 on dist-test-slave-bxm0"
server_key: "105c63a7292593434dfb82485db623e2"
server_key_iv: "b3149f64af88dada8d1735313a725f58"
server_key_version: "encryptionkey@0"
I20260927 21:30:00.955438 14002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-2/wal/instance:
uuid: "2003c5fc8ddf40498343730960161195"
format_stamp: "Formatted at 2026-09-27 21:30:00 on dist-test-slave-bxm0"
server_key: "105c63a7292593434dfb82485db623e2"
server_key_iv: "b3149f64af88dada8d1735313a725f58"
server_key_version: "encryptionkey@0"
I20260927 21:30:00.957015 13921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } attempt: 3
I20260927 21:30:00.958112 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.957598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43763 (local address 127.0.0.1:47648)
0927 21:30:00.957836 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.957927 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 21:30:00.959144 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:00.958780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43763 (local address 127.0.0.1:47652)
0927 21:30:00.958952 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:00.959037 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 21:30:00.959363 13921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43763: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43763: connect: Connection refused (error 111)
I20260927 21:30:00.961496 14002 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260927 21:30:00.965027 14017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:00.966423 14002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:30:00.966610 14002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-2/wal
uuid: "2003c5fc8ddf40498343730960161195"
format_stamp: "Formatted at 2026-09-27 21:30:00 on dist-test-slave-bxm0"
server_key: "105c63a7292593434dfb82485db623e2"
server_key_iv: "b3149f64af88dada8d1735313a725f58"
server_key_version: "encryptionkey@0"
I20260927 21:30:00.966832 14002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:30:01.007043 14002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:30:01.011214 14002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:30:01.011534 14002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:30:01.030934 14002 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:43763
I20260927 21:30:01.030957 14068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:43763 every 8 connection(s)
I20260927 21:30:01.032457 14002 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-2/data/info.pb
I20260927 21:30:01.035795 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 14002
I20260927 21:30:01.036005 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790544390748963-388-0/minicluster-data/master-2/wal/instance
I20260927 21:30:01.036404 14069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:30:01.036520 388 external_mini_cluster.cc:1428] Setting key 3a76498d030fb96967d1a862779c09c8
I20260927 21:30:01.046301 14069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41081 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:30:01 dist-test-slave-bxm0 krb5kdc[13841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 21:30:01 dist-test-slave-bxm0 krb5kdc[13841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:30:01.062366 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:01.038381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47664 (local address 127.0.97.60:43763)
0927 21:30:01.039291 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:01.039303 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:01.039348 (+ 45us) server_negotiation.cc:407] Connection header received
0927 21:30:01.041120 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:01.041145 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:01.041578 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:01.042021 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:01.043901 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.045190 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.047238 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.048012 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.048161 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:01.053271 (+ 5110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:01.053278 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:01.053289 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:01.053328 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:01.057190 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:01.057957 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:01.057967 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:01.057976 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:01.058204 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:01.058647 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:01.058650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:01.058652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:01.059255 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:01.059493 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:01.060424 (+ 931us) server_negotiation.cc:299] Negotiation successful
0927 21:30:01.060860 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":376,"threads_started":1}
I20260927 21:30:01.066232 14074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:01.049350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41081 (local address 127.0.0.1:47152)
0927 21:30:01.050352 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:01.050394 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:01.050545 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:01.050962 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:01.050967 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:01.051707 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:30:01.052112 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:01.052142 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.053554 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.053559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:01.055388 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:01.055409 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.056213 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.056219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:01.056395 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:01.057061 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:30:01.057090 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:30:01.059631 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:30:01.063378 (+ 3747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:01.063390 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:01.063399 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:01.063912 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:01.064343 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:01.064346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:01.064348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:01.064503 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:01.065093 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:30:01.065098 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:30:01.065252 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 21:30:01.065451 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 21:30:01.065808 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":593,"threads_started":1}
I20260927 21:30:01.066481 14073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:01.049372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47152 (local address 127.0.97.62:41081)
0927 21:30:01.049961 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:01.049969 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:01.050492 (+ 523us) server_negotiation.cc:407] Connection header received
0927 21:30:01.050680 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:01.050687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:01.050777 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:01.050903 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:01.052291 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.053406 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.055541 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.056021 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.056122 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:01.059873 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:01.059880 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:01.059885 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:01.059935 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:01.063238 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:01.064059 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:01.064064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:01.064068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:01.064231 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:01.064630 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:01.064635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:01.064638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:01.064952 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:01.065101 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:01.065589 (+ 488us) server_negotiation.cc:299] Negotiation successful
0927 21:30:01.065850 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":308,"threads_started":1}
I20260927 21:30:01.068682 14069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:30:01 dist-test-slave-bxm0 krb5kdc[13841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 27 21:30:01 dist-test-slave-bxm0 krb5kdc[13841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 21:30:01.082707 14075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:01.065817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47166 (local address 127.0.97.62:41081)
0927 21:30:01.066898 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:01.066918 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:01.066941 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:30:01.066978 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:01.066981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:01.067033 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:01.067116 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:01.068319 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.069438 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.071271 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.071915 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.072051 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:01.076520 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:01.076528 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:01.076533 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:01.076577 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:01.079425 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:01.080350 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:01.080354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:01.080356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:01.080534 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:01.080952 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:01.080958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:01.080961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:01.081334 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:01.081674 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:01.081933 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 21:30:01.082112 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":966,"thread_start_us":765,"threads_started":1}
I20260927 21:30:01.083284 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:01.069715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51602 (local address 127.0.97.61:33831)
0927 21:30:01.070016 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:01.070022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:01.070046 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:30:01.070149 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:01.070155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:01.070233 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:01.070334 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:01.071068 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.072123 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.073268 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.073794 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.073944 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:01.077007 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:01.077012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:01.077015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:01.077049 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:01.080344 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:01.081144 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:01.081148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:01.081150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:01.081298 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:01.081800 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:01.081809 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:01.081812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:01.082162 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:01.082317 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:01.082751 (+ 434us) server_negotiation.cc:299] Negotiation successful
0927 21:30:01.083011 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260927 21:30:01.083750 14074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:01.069656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33831 (local address 127.0.0.1:51602)
0927 21:30:01.069900 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:01.069916 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:01.070030 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:01.070353 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:01.070356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:01.070772 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:30:01.070915 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:01.070923 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.072262 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.072267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:01.073117 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:01.073128 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.073959 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.073966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:01.074157 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:01.074695 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:30:01.074712 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:30:01.076796 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:30:01.080498 (+ 3702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:01.080503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:01.080506 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:01.081019 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:01.081426 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:01.081434 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:01.081437 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:01.081663 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:01.082297 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:30:01.082303 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:30:01.082506 (+ 203us) client_negotiation.cc:769] Sending connection context
0927 21:30:01.082658 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 21:30:01.083400 (+ 742us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 21:30:01.085048 14069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:30:01 dist-test-slave-bxm0 krb5kdc[13841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 21:30:01 dist-test-slave-bxm0 krb5kdc[13841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 21:30:01.098407 14074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:01.085751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43763 (local address 127.0.0.1:47672)
0927 21:30:01.085890 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:01.085905 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:01.085979 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:01.086355 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:01.086359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:01.086863 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:30:01.087028 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:01.087039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.088106 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.088109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:01.088994 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:01.089009 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.089766 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.089770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:01.089951 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:01.090562 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:30:01.090578 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:30:01.092530 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:30:01.095794 (+ 3264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:01.095799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:01.095803 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:01.096290 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:01.096620 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:01.096628 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:01.096634 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:01.096796 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:01.097292 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:30:01.097299 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:30:01.097456 (+ 157us) client_negotiation.cc:769] Sending connection context
0927 21:30:01.097676 (+ 220us) client_negotiation.cc:240] Negotiation successful
0927 21:30:01.098002 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":22}
I20260927 21:30:01.098534 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:01.085993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47672 (local address 127.0.97.60:43763)
0927 21:30:01.086150 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:01.086154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:01.086166 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:30:01.086197 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:01.086200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:01.086246 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:01.086343 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:01.087177 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.087978 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.089127 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.089601 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.089751 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:01.092734 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:01.092739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:01.092742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:01.092770 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:01.095669 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:01.096405 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:01.096410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:01.096412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:01.096515 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:01.096908 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:01.096911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:01.096913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:01.097178 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:01.097287 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:01.097772 (+ 485us) server_negotiation.cc:299] Negotiation successful
0927 21:30:01.098234 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":221,"server-negotiator.queue_time_us":56}
I20260927 21:30:01.100116 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:01.084477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51608 (local address 127.0.97.61:33831)
0927 21:30:01.084630 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:01.084634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:01.084730 (+ 96us) server_negotiation.cc:407] Connection header received
0927 21:30:01.084936 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:01.084941 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:01.085011 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:01.085109 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:01.086618 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.087572 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.088935 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.089437 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.089528 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:01.094031 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:01.094036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:01.094039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:01.094070 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:01.097021 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:01.097885 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:01.097891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:01.097895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:01.098065 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:01.098642 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:01.098648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:01.098654 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:01.098982 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:01.099225 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:01.099557 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 21:30:01.099783 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:30:01.104532 14069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2003c5fc8ddf40498343730960161195: Bootstrap starting.
I20260927 21:30:01.104887 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:30:01.107918 14069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2003c5fc8ddf40498343730960161195: Neither blocks nor log segments found. Creating new log.
I20260927 21:30:01.108946 14069 log.cc:824] T 00000000000000000000000000000000 P 2003c5fc8ddf40498343730960161195: Log is configured to *not* fsync() on all Append() calls
I20260927 21:30:01.112675 14069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2003c5fc8ddf40498343730960161195: No bootstrap required, opened a new log
I20260927 21:30:01.124181 14069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2003c5fc8ddf40498343730960161195 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41081 } } peers { permanent_uuid: "8fe1e1831c78480093f9d4b7b0dcedc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 } } peers { permanent_uuid: "2003c5fc8ddf40498343730960161195" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } }
I20260927 21:30:01.124672 14069 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2003c5fc8ddf40498343730960161195 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:30:01.124805 14069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2003c5fc8ddf40498343730960161195 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2003c5fc8ddf40498343730960161195, State: Initialized, Role: FOLLOWER
I20260927 21:30:01.126117 14069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2003c5fc8ddf40498343730960161195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41081 } } peers { permanent_uuid: "8fe1e1831c78480093f9d4b7b0dcedc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 } } peers { permanent_uuid: "2003c5fc8ddf40498343730960161195" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } }
I20260927 21:30:01.132614 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:01.110112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51618 (local address 127.0.97.61:33831)
0927 21:30:01.110328 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:01.110337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:01.114190 (+ 3853us) server_negotiation.cc:407] Connection header received
0927 21:30:01.114469 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:01.114476 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:01.114554 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:01.114624 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:01.116792 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.117925 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.120360 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.121014 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.121112 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:01.125165 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:01.125175 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:01.125181 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:01.125230 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:01.129197 (+ 3967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:01.130164 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:01.130170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:01.130174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:01.130341 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:01.130808 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:01.130814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:01.130818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:01.131187 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:01.131418 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:01.132000 (+ 582us) server_negotiation.cc:299] Negotiation successful
0927 21:30:01.132249 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 21:30:01.136581 13921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } attempt: 4
I20260927 21:30:01.136179 14084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2003c5fc8ddf40498343730960161195 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41081 } } peers { permanent_uuid: "8fe1e1831c78480093f9d4b7b0dcedc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 } } peers { permanent_uuid: "2003c5fc8ddf40498343730960161195" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } } }
I20260927 21:30:01.136883 14075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:01.110474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47178 (local address 127.0.97.62:41081)
0927 21:30:01.110665 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:01.110669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:01.113280 (+ 2611us) server_negotiation.cc:407] Connection header received
0927 21:30:01.113492 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:01.113501 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:01.113639 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:01.113761 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:01.116550 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.118050 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.121073 (+ 3023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.122018 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.122148 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:01.126527 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:01.126533 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:01.126540 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:01.126587 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:01.131426 (+ 4839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:01.132550 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:01.132559 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:01.132565 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:01.132736 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:01.133319 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:01.133327 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:01.133334 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:01.135398 (+ 2064us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:01.135723 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:01.136266 (+ 543us) server_negotiation.cc:299] Negotiation successful
0927 21:30:01.136492 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:30:01.136893 14084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2003c5fc8ddf40498343730960161195 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:30:01.138124 14069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2003c5fc8ddf40498343730960161195 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:30:01.144914 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:01.114325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47678 (local address 127.0.97.60:43763)
0927 21:30:01.114545 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:01.114553 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:01.125833 (+ 11280us) server_negotiation.cc:407] Connection header received
0927 21:30:01.126085 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:01.126090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:01.126173 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:01.126282 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:01.128289 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.129453 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.131567 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.132347 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.132459 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:01.135654 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:01.135662 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:01.135669 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:01.135720 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:01.140931 (+ 5211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:01.142071 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:01.142080 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:01.142086 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:01.142319 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:01.142917 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:01.142925 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:01.142935 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:01.143405 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:01.143695 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:01.144295 (+ 600us) server_negotiation.cc:299] Negotiation successful
0927 21:30:01.144525 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 21:30:01 dist-test-slave-bxm0 krb5kdc[13841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260927 21:30:01.158015 14096 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2003c5fc8ddf40498343730960161195: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:30:01.158161 14096 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:30:01.159053 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:01.137184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43763 (local address 127.0.0.1:47680)
0927 21:30:01.137358 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:01.137376 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:01.137484 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:01.142859 (+ 5375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:01.142866 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:01.143408 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:30:01.143638 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:01.143654 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.145198 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.145204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:01.146630 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:01.146643 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.147900 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.147905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:01.148036 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:01.148711 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:30:01.148728 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:30:01.151381 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:30:01.155827 (+ 4446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:01.155832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:01.155836 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:01.156348 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:01.156905 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:01.156912 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:01.156918 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:01.157106 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:01.157996 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:30:01.158002 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:30:01.158167 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 21:30:01.158318 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 21:30:01.158642 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 21:30:01.159144 14085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:01.137572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47680 (local address 127.0.97.60:43763)
0927 21:30:01.142579 (+ 5007us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:01.142583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:01.142601 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:30:01.142639 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:01.142642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:01.142699 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:01.142820 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:01.143799 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.145024 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.146775 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.147740 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.147917 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:01.151657 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:01.151667 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:01.151674 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:01.151714 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:01.155609 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:01.156539 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:01.156544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:01.156547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:01.156733 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:01.157241 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:01.157251 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:01.157255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:01.157766 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:01.157932 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:01.158426 (+ 494us) server_negotiation.cc:299] Negotiation successful
0927 21:30:01.158665 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4887,"thread_start_us":4723,"threads_started":1}
I20260927 21:30:01.153184 14002 master_runner.cc:186] Error getting master registration for 127.0.97.62:41081: OK, instance_id { permanent_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" instance_seqno: 1790544599993883 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:30:01.164722 14002 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:30:01.164983 13921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d: Bootstrap starting.
I20260927 21:30:01.168617 13921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d: Neither blocks nor log segments found. Creating new log.
I20260927 21:30:01.169762 13921 log.cc:824] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d: Log is configured to *not* fsync() on all Append() calls
I20260927 21:30:01.172866 13921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d: No bootstrap required, opened a new log
I20260927 21:30:01.181027 13921 raft_consensus.cc:348] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41081 } } peers { permanent_uuid: "8fe1e1831c78480093f9d4b7b0dcedc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 } } peers { permanent_uuid: "2003c5fc8ddf40498343730960161195" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } }
I20260927 21:30:01.181349 13921 raft_consensus.cc:374] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:30:01.181495 13921 raft_consensus.cc:729] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 860cf650a2aa4fe69323f0fdd41ce16d, State: Initialized, Role: FOLLOWER
I20260927 21:30:01.182127 13921 consensus_queue.cc:260] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41081 } } peers { permanent_uuid: "8fe1e1831c78480093f9d4b7b0dcedc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 } } peers { permanent_uuid: "2003c5fc8ddf40498343730960161195" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } }
I20260927 21:30:01.183135 14099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41081 } } peers { permanent_uuid: "8fe1e1831c78480093f9d4b7b0dcedc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 } } peers { permanent_uuid: "2003c5fc8ddf40498343730960161195" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } } }
I20260927 21:30:01.183601 14099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:30:01.184283 13921 sys_catalog.cc:564] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [sys.catalog]: configured and running, proceeding with master startup.
W20260927 21:30:01.195585 14110 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:30:01.195848 14110 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:30:01.195004 13854 master_runner.cc:186] Error getting master registration for 127.0.97.61:33831: OK, instance_id { permanent_uuid: "8fe1e1831c78480093f9d4b7b0dcedc1" instance_seqno: 1790544600496367 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:30:01.203790 13854 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:30:01.228498 13995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } attempt: 5
W20260927 21:30:01.233415 13875 tablet.cc:2369] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 21:30:01 dist-test-slave-bxm0 krb5kdc[13841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 21:30:01.243757 14085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:01.229233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47686 (local address 127.0.97.60:43763)
0927 21:30:01.229419 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:01.229425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:01.229546 (+ 121us) server_negotiation.cc:407] Connection header received
0927 21:30:01.229794 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:01.229801 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:01.229887 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:01.230009 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:01.230887 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.231841 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.233581 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.234281 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.234402 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:01.237886 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:01.237892 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:01.237897 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:01.237937 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:01.241038 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:01.241740 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:01.241746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:01.241750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:01.241886 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:01.242276 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:01.242280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:01.242283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:01.242565 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:01.242725 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:01.243121 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 21:30:01.243342 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:30:01.243799 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:01.229167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43763 (local address 127.0.0.1:47686)
0927 21:30:01.229439 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:01.229459 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:01.229658 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:01.230035 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:01.230040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:01.230558 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:30:01.230715 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:01.230725 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.232027 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.232032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:01.233418 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:01.233432 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:01.234620 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:01.234627 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:01.234812 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:01.235509 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:30:01.235526 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:30:01.237693 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:30:01.241188 (+ 3495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:01.241195 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:01.241199 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:01.241591 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:01.241998 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:01.242006 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:01.242010 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:01.242153 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:01.242686 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:30:01.242691 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:30:01.242849 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 21:30:01.242999 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 21:30:01.243341 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260927 21:30:01.248858 13995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fe1e1831c78480093f9d4b7b0dcedc1: Bootstrap starting.
I20260927 21:30:01.252954 13995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fe1e1831c78480093f9d4b7b0dcedc1: Neither blocks nor log segments found. Creating new log.
I20260927 21:30:01.254230 13995 log.cc:824] T 00000000000000000000000000000000 P 8fe1e1831c78480093f9d4b7b0dcedc1: Log is configured to *not* fsync() on all Append() calls
I20260927 21:30:01.257126 13995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fe1e1831c78480093f9d4b7b0dcedc1: No bootstrap required, opened a new log
W20260927 21:30:01.263698 14023 tablet.cc:2369] T 00000000000000000000000000000000 P 2003c5fc8ddf40498343730960161195: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:30:01.265595 13995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fe1e1831c78480093f9d4b7b0dcedc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41081 } } peers { permanent_uuid: "8fe1e1831c78480093f9d4b7b0dcedc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 } } peers { permanent_uuid: "2003c5fc8ddf40498343730960161195" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } }
I20260927 21:30:01.266047 13995 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fe1e1831c78480093f9d4b7b0dcedc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:30:01.266199 13995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fe1e1831c78480093f9d4b7b0dcedc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fe1e1831c78480093f9d4b7b0dcedc1, State: Initialized, Role: FOLLOWER
I20260927 21:30:01.266851 13995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fe1e1831c78480093f9d4b7b0dcedc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41081 } } peers { permanent_uuid: "8fe1e1831c78480093f9d4b7b0dcedc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 } } peers { permanent_uuid: "2003c5fc8ddf40498343730960161195" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } }
I20260927 21:30:01.273904 14113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fe1e1831c78480093f9d4b7b0dcedc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41081 } } peers { permanent_uuid: "8fe1e1831c78480093f9d4b7b0dcedc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 } } peers { permanent_uuid: "2003c5fc8ddf40498343730960161195" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } } }
I20260927 21:30:01.274264 14113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fe1e1831c78480093f9d4b7b0dcedc1 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:30:01.275655 13995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fe1e1831c78480093f9d4b7b0dcedc1 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 21:30:01.296150 14124 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8fe1e1831c78480093f9d4b7b0dcedc1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:30:01.296422 14124 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:30:01.296643 13928 master_runner.cc:263] No leader master found from master 8fe1e1831c78480093f9d4b7b0dcedc1
I20260927 21:30:01.296792 13928 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:30:01.301924 14099 raft_consensus.cc:482] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:30:01.302218 14099 raft_consensus.cc:504] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41081 } } peers { permanent_uuid: "8fe1e1831c78480093f9d4b7b0dcedc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 } } peers { permanent_uuid: "2003c5fc8ddf40498343730960161195" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } }
I20260927 21:30:01.303823 14099 leader_election.cc:288] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fe1e1831c78480093f9d4b7b0dcedc1 (127.0.97.61:33831), 2003c5fc8ddf40498343730960161195 (127.0.97.60:43763)
I20260927 21:30:01.305027 13970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fe1e1831c78480093f9d4b7b0dcedc1" is_pre_election: true
I20260927 21:30:01.306447 14044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2003c5fc8ddf40498343730960161195" is_pre_election: true
I20260927 21:30:01.307112 14044 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2003c5fc8ddf40498343730960161195 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 860cf650a2aa4fe69323f0fdd41ce16d in term 0.
I20260927 21:30:01.308950 13970 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8fe1e1831c78480093f9d4b7b0dcedc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 860cf650a2aa4fe69323f0fdd41ce16d in term 0.
I20260927 21:30:01.309794 13871 leader_election.cc:302] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 860cf650a2aa4fe69323f0fdd41ce16d, 8fe1e1831c78480093f9d4b7b0dcedc1; no voters:
I20260927 21:30:01.310832 14099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:30:01.310992 14099 raft_consensus.cc:482] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:30:01.311095 14099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:30:01.314697 14099 raft_consensus.cc:504] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41081 } } peers { permanent_uuid: "8fe1e1831c78480093f9d4b7b0dcedc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 } } peers { permanent_uuid: "2003c5fc8ddf40498343730960161195" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } }
I20260927 21:30:01.315518 14099 leader_election.cc:288] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [CANDIDATE]: Term 1 election: Requested vote from peers 8fe1e1831c78480093f9d4b7b0dcedc1 (127.0.97.61:33831), 2003c5fc8ddf40498343730960161195 (127.0.97.60:43763)
I20260927 21:30:01.319594 13970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fe1e1831c78480093f9d4b7b0dcedc1"
I20260927 21:30:01.319846 13970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fe1e1831c78480093f9d4b7b0dcedc1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:30:01.323472 13970 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8fe1e1831c78480093f9d4b7b0dcedc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 860cf650a2aa4fe69323f0fdd41ce16d in term 1.
I20260927 21:30:01.324143 14044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2003c5fc8ddf40498343730960161195"
I20260927 21:30:01.324399 14044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2003c5fc8ddf40498343730960161195 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:30:01.327926 14044 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2003c5fc8ddf40498343730960161195 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 860cf650a2aa4fe69323f0fdd41ce16d in term 1.
I20260927 21:30:01.328590 13871 leader_election.cc:302] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2003c5fc8ddf40498343730960161195, 860cf650a2aa4fe69323f0fdd41ce16d; no voters:
I20260927 21:30:01.329255 14099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:30:01.339563 14099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [term 1 LEADER]: Becoming Leader. State: Replica: 860cf650a2aa4fe69323f0fdd41ce16d, State: Running, Role: LEADER
I20260927 21:30:01.340386 14099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41081 } } peers { permanent_uuid: "8fe1e1831c78480093f9d4b7b0dcedc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 } } peers { permanent_uuid: "2003c5fc8ddf40498343730960161195" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } }
I20260927 21:30:01.354508 14126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 860cf650a2aa4fe69323f0fdd41ce16d. Latest consensus state: current_term: 1 leader_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41081 } } peers { permanent_uuid: "8fe1e1831c78480093f9d4b7b0dcedc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 } } peers { permanent_uuid: "2003c5fc8ddf40498343730960161195" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } } }
I20260927 21:30:01.355177 14126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [sys.catalog]: This master's current role is: LEADER
I20260927 21:30:01.364200 14128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:30:01.366812 14128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:30:01.387996 14044 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2003c5fc8ddf40498343730960161195 [term 1 FOLLOWER]: Refusing update from remote peer 860cf650a2aa4fe69323f0fdd41ce16d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:30:01.389359 14126 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2003c5fc8ddf40498343730960161195" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:30:01.400630 13970 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8fe1e1831c78480093f9d4b7b0dcedc1 [term 1 FOLLOWER]: Refusing update from remote peer 860cf650a2aa4fe69323f0fdd41ce16d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:30:01.401700 14126 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fe1e1831c78480093f9d4b7b0dcedc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:30:01.436530 14113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fe1e1831c78480093f9d4b7b0dcedc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 860cf650a2aa4fe69323f0fdd41ce16d. Latest consensus state: current_term: 1 leader_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41081 } } peers { permanent_uuid: "8fe1e1831c78480093f9d4b7b0dcedc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 } } peers { permanent_uuid: "2003c5fc8ddf40498343730960161195" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } } }
I20260927 21:30:01.436887 14113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fe1e1831c78480093f9d4b7b0dcedc1 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:30:01.442916 14129 mvcc.cc:205] Tried to move back new op lower bound from 7334070687270526976 to 7334070687097102336. Current Snapshot: MvccSnapshot[applied={T|T < 7334070687270526976}]
I20260927 21:30:01.443929 14131 mvcc.cc:205] Tried to move back new op lower bound from 7334070687270526976 to 7334070687097102336. Current Snapshot: MvccSnapshot[applied={T|T < 7334070687270526976}]
I20260927 21:30:01.449280 14084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2003c5fc8ddf40498343730960161195 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 860cf650a2aa4fe69323f0fdd41ce16d. Latest consensus state: current_term: 1 leader_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41081 } } peers { permanent_uuid: "8fe1e1831c78480093f9d4b7b0dcedc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 } } peers { permanent_uuid: "2003c5fc8ddf40498343730960161195" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } } }
I20260927 21:30:01.451121 14084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2003c5fc8ddf40498343730960161195 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:30:01.454003 14113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fe1e1831c78480093f9d4b7b0dcedc1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41081 } } peers { permanent_uuid: "8fe1e1831c78480093f9d4b7b0dcedc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 } } peers { permanent_uuid: "2003c5fc8ddf40498343730960161195" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } } }
I20260927 21:30:01.454304 14113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fe1e1831c78480093f9d4b7b0dcedc1 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:30:01.454160 14130 mvcc.cc:205] Tried to move back new op lower bound from 7334070687270526976 to 7334070687097102336. Current Snapshot: MvccSnapshot[applied={T|T < 7334070687270526976}]
I20260927 21:30:01.457383 14099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41081 } } peers { permanent_uuid: "8fe1e1831c78480093f9d4b7b0dcedc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 } } peers { permanent_uuid: "2003c5fc8ddf40498343730960161195" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } } }
I20260927 21:30:01.457780 14099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [sys.catalog]: This master's current role is: LEADER
I20260927 21:30:01.458175 14099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41081 } } peers { permanent_uuid: "8fe1e1831c78480093f9d4b7b0dcedc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 } } peers { permanent_uuid: "2003c5fc8ddf40498343730960161195" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } } }
I20260927 21:30:01.458425 14099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d [sys.catalog]: This master's current role is: LEADER
I20260927 21:30:01.461510 14128 catalog_manager.cc:1347] Generated new cluster ID: 8a8c7d4b7d4d44e58b81d95e6c6add62
I20260927 21:30:01.461751 14128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:30:01.473773 14084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2003c5fc8ddf40498343730960161195 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "860cf650a2aa4fe69323f0fdd41ce16d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41081 } } peers { permanent_uuid: "8fe1e1831c78480093f9d4b7b0dcedc1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33831 } } peers { permanent_uuid: "2003c5fc8ddf40498343730960161195" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43763 } } }
I20260927 21:30:01.474102 14084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2003c5fc8ddf40498343730960161195 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:30:01.511756 14128 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:30:01.513921 14128 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:30:01.534089 14128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 860cf650a2aa4fe69323f0fdd41ce16d: Generated new TSK 0
I20260927 21:30:01.535204 14128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:30:02.161118 14096 catalog_manager.cc:1259] Loaded cluster ID: 8a8c7d4b7d4d44e58b81d95e6c6add62
I20260927 21:30:02.161262 14096 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2003c5fc8ddf40498343730960161195: loading cluster ID for follower catalog manager: success
I20260927 21:30:02.165659 14096 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2003c5fc8ddf40498343730960161195: acquiring CA information for follower catalog manager: success
I20260927 21:30:02.168634 14096 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2003c5fc8ddf40498343730960161195: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:30:02.298560 14124 catalog_manager.cc:1259] Loaded cluster ID: 8a8c7d4b7d4d44e58b81d95e6c6add62
I20260927 21:30:02.298668 14124 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8fe1e1831c78480093f9d4b7b0dcedc1: loading cluster ID for follower catalog manager: success
I20260927 21:30:02.301136 14124 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8fe1e1831c78480093f9d4b7b0dcedc1: acquiring CA information for follower catalog manager: success
I20260927 21:30:02.303059 14124 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8fe1e1831c78480093f9d4b7b0dcedc1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:30:02.693987 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 13854
I20260927 21:30:02.695257 13916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:30:02.808583 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 13854
I20260927 21:30:02.829814 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 13928
I20260927 21:30:02.830977 13990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:30:02.937995 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 13928
I20260927 21:30:02.958422 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 14002
I20260927 21:30:02.959587 14064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:30:03.067571 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 14002
2026-09-27T21:30:03Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4868 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:30:03 dist-test-slave-bxm0 krb5kdc[14149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:30:03 dist-test-slave-bxm0 krb5kdc[14149](info): set up 2 sockets
Sep 27 21:30:03 dist-test-slave-bxm0 krb5kdc[14149](info): commencing operation
krb5kdc: starting...
W20260927 21:30:04.178174 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:30:04 dist-test-slave-bxm0 krb5kdc[14149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:30:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:30:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:30:04.418231 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37519
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:30:04.754215 14162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:30:04.754539 14162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:30:04.754629 14162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:30:04.761961 14162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:30:04.762079 14162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:30:04.762130 14162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:30:04.762188 14162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:30:04.762224 14162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:30:04.772204 14162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:30:04.773517 14162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:30:04.774797 14162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:30:04.786443 14167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:04.786455 14168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:04.786480 14170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:30:04.787596 14162 server_base.cc:1034] running on GCE node
I20260927 21:30:04.788395 14162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:30:04.789932 14162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:30:04.791168 14162 hybrid_clock.cc:648] HybridClock initialized: now 1790544604791130 us; error 50 us; skew 500 ppm
I20260927 21:30:04.791520 14162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:30:04 dist-test-slave-bxm0 krb5kdc[14149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:30:04.798153 14162 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:30:04.799646 14162 webserver.cc:460] Webserver started at http://127.0.97.62:46153/ using document root <none> and password file <none>
I20260927 21:30:04.800295 14162 fs_manager.cc:359] Metadata directory not provided
I20260927 21:30:04.800390 14162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:30:04.800658 14162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:30:04.803742 14162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "09343c8fa732403b8274271d0b72bf50"
format_stamp: "Formatted at 2026-09-27 21:30:04 on dist-test-slave-bxm0"
server_key: "fe934ccb3521f3854e9180f0d8764753"
server_key_iv: "4cb08f0b0ef60e8f07c00ba6eb9f32c2"
server_key_version: "encryptionkey@0"
I20260927 21:30:04.804447 14162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "09343c8fa732403b8274271d0b72bf50"
format_stamp: "Formatted at 2026-09-27 21:30:04 on dist-test-slave-bxm0"
server_key: "fe934ccb3521f3854e9180f0d8764753"
server_key_iv: "4cb08f0b0ef60e8f07c00ba6eb9f32c2"
server_key_version: "encryptionkey@0"
I20260927 21:30:04.809754 14162 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260927 21:30:04.813107 14177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:04.814479 14162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:30:04.814671 14162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "09343c8fa732403b8274271d0b72bf50"
format_stamp: "Formatted at 2026-09-27 21:30:04 on dist-test-slave-bxm0"
server_key: "fe934ccb3521f3854e9180f0d8764753"
server_key_iv: "4cb08f0b0ef60e8f07c00ba6eb9f32c2"
server_key_version: "encryptionkey@0"
I20260927 21:30:04.814890 14162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:30:04.855381 14162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:30:04.859629 14162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:30:04.859932 14162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:30:04.879069 14162 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37519
I20260927 21:30:04.879119 14228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37519 every 8 connection(s)
I20260927 21:30:04.880517 14162 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:30:04.881222 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 14162
I20260927 21:30:04.881492 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:30:04.882318 388 external_mini_cluster.cc:1428] Setting key d4b966e11f0bd9af64bbaadaf25c6d79
I20260927 21:30:04.884886 14229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 21:30:04 dist-test-slave-bxm0 krb5kdc[14149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:30:04.898386 14229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09343c8fa732403b8274271d0b72bf50: Bootstrap starting.
I20260927 21:30:04.902611 14229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09343c8fa732403b8274271d0b72bf50: Neither blocks nor log segments found. Creating new log.
I20260927 21:30:04.903807 14229 log.cc:824] T 00000000000000000000000000000000 P 09343c8fa732403b8274271d0b72bf50: Log is configured to *not* fsync() on all Append() calls
I20260927 21:30:04.907317 14232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:04.884270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50516 (local address 127.0.97.62:37519)
0927 21:30:04.885095 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:04.885106 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:04.885144 (+ 38us) server_negotiation.cc:407] Connection header received
0927 21:30:04.886834 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:04.886856 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:04.887219 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:04.887560 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:04.889419 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:04.890889 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:04.892771 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:04.893478 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:04.893702 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:04.898634 (+ 4932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:04.898644 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:04.898664 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:04.898709 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:04.902685 (+ 3976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:04.903407 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:04.903416 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:04.903424 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:04.903657 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:04.904019 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:04.904023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:04.904026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:04.904753 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:04.904969 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:04.905569 (+ 600us) server_negotiation.cc:299] Negotiation successful
0927 21:30:04.906008 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":356,"threads_started":1}
I20260927 21:30:04.907402 14229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09343c8fa732403b8274271d0b72bf50: No bootstrap required, opened a new log
I20260927 21:30:04.911170 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:30:04.916329 14229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09343c8fa732403b8274271d0b72bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09343c8fa732403b8274271d0b72bf50" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37519 } }
I20260927 21:30:04.916741 14229 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09343c8fa732403b8274271d0b72bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:30:04.916872 14229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09343c8fa732403b8274271d0b72bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09343c8fa732403b8274271d0b72bf50, State: Initialized, Role: FOLLOWER
I20260927 21:30:04.917512 14229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09343c8fa732403b8274271d0b72bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09343c8fa732403b8274271d0b72bf50" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37519 } }
I20260927 21:30:04.917841 14229 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09343c8fa732403b8274271d0b72bf50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:30:04.918006 14229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09343c8fa732403b8274271d0b72bf50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:30:04.918200 14229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09343c8fa732403b8274271d0b72bf50 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:30:04.920397 14229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09343c8fa732403b8274271d0b72bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09343c8fa732403b8274271d0b72bf50" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37519 } }
I20260927 21:30:04.920904 14229 leader_election.cc:302] T 00000000000000000000000000000000 P 09343c8fa732403b8274271d0b72bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09343c8fa732403b8274271d0b72bf50; no voters:
I20260927 21:30:04.921542 14229 leader_election.cc:288] T 00000000000000000000000000000000 P 09343c8fa732403b8274271d0b72bf50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:30:04.921777 14234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09343c8fa732403b8274271d0b72bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:30:04.922446 14234 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09343c8fa732403b8274271d0b72bf50 [term 1 LEADER]: Becoming Leader. State: Replica: 09343c8fa732403b8274271d0b72bf50, State: Running, Role: LEADER
I20260927 21:30:04.922992 14234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09343c8fa732403b8274271d0b72bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09343c8fa732403b8274271d0b72bf50" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37519 } }
I20260927 21:30:04.923739 14229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09343c8fa732403b8274271d0b72bf50 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:30:04.925938 14235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09343c8fa732403b8274271d0b72bf50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09343c8fa732403b8274271d0b72bf50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09343c8fa732403b8274271d0b72bf50" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37519 } } }
I20260927 21:30:04.926364 14235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09343c8fa732403b8274271d0b72bf50 [sys.catalog]: This master's current role is: LEADER
I20260927 21:30:04.926203 14236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09343c8fa732403b8274271d0b72bf50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09343c8fa732403b8274271d0b72bf50. Latest consensus state: current_term: 1 leader_uuid: "09343c8fa732403b8274271d0b72bf50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09343c8fa732403b8274271d0b72bf50" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37519 } } }
I20260927 21:30:04.926599 14236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09343c8fa732403b8274271d0b72bf50 [sys.catalog]: This master's current role is: LEADER
I20260927 21:30:04.930279 14240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:30:04.935860 14240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:30:04.944748 14240 catalog_manager.cc:1347] Generated new cluster ID: f34f585657e5494289f47834664d72a8
I20260927 21:30:04.944868 14240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:30:04.970589 14240 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:30:04.971956 14240 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:30:04.980873 14240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09343c8fa732403b8274271d0b72bf50: Generated new TSK 0
I20260927 21:30:04.981989 14240 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:30:05.023202 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 14162
I20260927 21:30:05.024480 14224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:30:05.120524 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 14162
2026-09-27T21:30:05Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2049 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:30:05 dist-test-slave-bxm0 krb5kdc[14257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:30:05 dist-test-slave-bxm0 krb5kdc[14257](info): set up 2 sockets
Sep 27 21:30:05 dist-test-slave-bxm0 krb5kdc[14257](info): commencing operation
krb5kdc: starting...
W20260927 21:30:06.218056 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:30:06 dist-test-slave-bxm0 krb5kdc[14257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:30:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:30:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:30:06.456920 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37797
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:30:06.787482 14270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:30:06.787803 14270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:30:06.787899 14270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:30:06.795286 14270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:30:06.795418 14270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:30:06.795465 14270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:30:06.795524 14270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:30:06.795562 14270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:30:06.805720 14270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:30:06.807044 14270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:30:06.808354 14270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:30:06.820137 14276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:06.820132 14278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:06.820647 14275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:30:06.820820 14270 server_base.cc:1034] running on GCE node
I20260927 21:30:06.821684 14270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:30:06.823141 14270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:30:06.824357 14270 hybrid_clock.cc:648] HybridClock initialized: now 1790544606824317 us; error 27 us; skew 500 ppm
I20260927 21:30:06.824668 14270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:30:06 dist-test-slave-bxm0 krb5kdc[14257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:30:06.827849 14270 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:30:06.829339 14270 webserver.cc:460] Webserver started at http://127.0.97.62:40891/ using document root <none> and password file <none>
I20260927 21:30:06.830103 14270 fs_manager.cc:359] Metadata directory not provided
I20260927 21:30:06.830231 14270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:30:06.830559 14270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:30:06.833557 14270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "76f94b335cdc416d9b1742c8e36dc751"
format_stamp: "Formatted at 2026-09-27 21:30:06 on dist-test-slave-bxm0"
server_key: "7f05210ad6401a58ef0ee35f900a0a14"
server_key_iv: "677e47646ef496327190ebca3c0fb114"
server_key_version: "encryptionkey@0"
I20260927 21:30:06.834389 14270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "76f94b335cdc416d9b1742c8e36dc751"
format_stamp: "Formatted at 2026-09-27 21:30:06 on dist-test-slave-bxm0"
server_key: "7f05210ad6401a58ef0ee35f900a0a14"
server_key_iv: "677e47646ef496327190ebca3c0fb114"
server_key_version: "encryptionkey@0"
I20260927 21:30:06.839690 14270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:30:06.843191 14285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:06.844523 14270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:30:06.844707 14270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "76f94b335cdc416d9b1742c8e36dc751"
format_stamp: "Formatted at 2026-09-27 21:30:06 on dist-test-slave-bxm0"
server_key: "7f05210ad6401a58ef0ee35f900a0a14"
server_key_iv: "677e47646ef496327190ebca3c0fb114"
server_key_version: "encryptionkey@0"
I20260927 21:30:06.844924 14270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:30:06.862149 14270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:30:06.866226 14270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:30:06.866513 14270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:30:06.885057 14270 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37797
I20260927 21:30:06.885090 14336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37797 every 8 connection(s)
I20260927 21:30:06.886552 14270 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:30:06.889815 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 14270
I20260927 21:30:06.890038 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:30:06.890368 14337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:30:06.890753 388 external_mini_cluster.cc:1428] Setting key 552f0b20fc6a3072c524c975ba20203e
I20260927 21:30:06.904814 14337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751: Bootstrap starting.
Sep 27 21:30:06 dist-test-slave-bxm0 krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:30:06.908910 14337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751: Neither blocks nor log segments found. Creating new log.
I20260927 21:30:06.910346 14337 log.cc:824] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751: Log is configured to *not* fsync() on all Append() calls
I20260927 21:30:06.913913 14337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751: No bootstrap required, opened a new log
I20260927 21:30:06.915094 14340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:06.892781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53932 (local address 127.0.97.62:37797)
0927 21:30:06.893601 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:06.893634 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:06.893681 (+ 47us) server_negotiation.cc:407] Connection header received
0927 21:30:06.895290 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:06.895318 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:06.895811 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:06.896271 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:06.897477 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:06.899564 (+ 2087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:06.900851 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:06.902026 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:06.902217 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:06.906196 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:06.906207 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:06.906227 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:06.906278 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:06.910376 (+ 4098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:06.911279 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:06.911285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:06.911291 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:06.911456 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:06.911899 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:06.911904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:06.911906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:06.912590 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:06.912872 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:06.913274 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 21:30:06.913736 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":351,"threads_started":1}
I20260927 21:30:06.922273 14337 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f94b335cdc416d9b1742c8e36dc751" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37797 } }
I20260927 21:30:06.922681 14337 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:30:06.922796 14337 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f94b335cdc416d9b1742c8e36dc751, State: Initialized, Role: FOLLOWER
I20260927 21:30:06.923350 14337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f94b335cdc416d9b1742c8e36dc751" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37797 } }
I20260927 21:30:06.923594 14337 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:30:06.923738 14337 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:30:06.923902 14337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:30:06.926137 14337 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f94b335cdc416d9b1742c8e36dc751" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37797 } }
I20260927 21:30:06.926641 14337 leader_election.cc:302] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76f94b335cdc416d9b1742c8e36dc751; no voters:
I20260927 21:30:06.927259 14337 leader_election.cc:288] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:30:06.927415 14342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:30:06.928122 14342 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [term 1 LEADER]: Becoming Leader. State: Replica: 76f94b335cdc416d9b1742c8e36dc751, State: Running, Role: LEADER
I20260927 21:30:06.928691 14342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f94b335cdc416d9b1742c8e36dc751" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37797 } }
I20260927 21:30:06.929523 14337 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:30:06.931878 14344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76f94b335cdc416d9b1742c8e36dc751. Latest consensus state: current_term: 1 leader_uuid: "76f94b335cdc416d9b1742c8e36dc751" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f94b335cdc416d9b1742c8e36dc751" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37797 } } }
I20260927 21:30:06.931738 14343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76f94b335cdc416d9b1742c8e36dc751" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f94b335cdc416d9b1742c8e36dc751" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37797 } } }
I20260927 21:30:06.932468 14343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [sys.catalog]: This master's current role is: LEADER
I20260927 21:30:06.933061 14344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [sys.catalog]: This master's current role is: LEADER
I20260927 21:30:06.934141 14350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:30:06.939570 14350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:30:06.949517 14350 catalog_manager.cc:1347] Generated new cluster ID: 707f39afb8714b80845de7fb01a75a3f
I20260927 21:30:06.949661 14350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:30:06.976310 14350 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:30:06.978405 14350 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:30:06.993911 14350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751: Generated new TSK 0
I20260927 21:30:06.995048 14350 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 21:30:07.067413 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37797
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:30:07.397115 14363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:30:07.397469 14363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:30:07.397673 14363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:30:07.404982 14363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:30:07.405150 14363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:30:07.405342 14363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:30:07.415518 14363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:30:07.416882 14363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:30:07.418309 14363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:30:07.432125 14371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:07.432217 14369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:07.432446 14368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:30:07.433260 14363 server_base.cc:1034] running on GCE node
I20260927 21:30:07.433949 14363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:30:07.434890 14363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:30:07.436128 14363 hybrid_clock.cc:648] HybridClock initialized: now 1790544607436071 us; error 51 us; skew 500 ppm
I20260927 21:30:07.436448 14363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:30:07 dist-test-slave-bxm0 krb5kdc[14257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:30:07.439670 14363 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 21:30:07.441330 14363 webserver.cc:460] Webserver started at http://127.0.97.1:42881/ using document root <none> and password file <none>
I20260927 21:30:07.442066 14363 fs_manager.cc:359] Metadata directory not provided
I20260927 21:30:07.442209 14363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:30:07.442512 14363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:30:07.445402 14363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/data/instance:
uuid: "f92ecd52d3014aeabccda938a17e7e68"
format_stamp: "Formatted at 2026-09-27 21:30:07 on dist-test-slave-bxm0"
server_key: "55d24bccfd8ecfcbe8cc905254bc434b"
server_key_iv: "fc1e842703c8bf9739f43cfa3bae52e7"
server_key_version: "encryptionkey@0"
I20260927 21:30:07.446365 14363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f92ecd52d3014aeabccda938a17e7e68"
format_stamp: "Formatted at 2026-09-27 21:30:07 on dist-test-slave-bxm0"
server_key: "55d24bccfd8ecfcbe8cc905254bc434b"
server_key_iv: "fc1e842703c8bf9739f43cfa3bae52e7"
server_key_version: "encryptionkey@0"
I20260927 21:30:07.451936 14363 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:30:07.455508 14378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:07.456837 14363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:30:07.457028 14363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "f92ecd52d3014aeabccda938a17e7e68"
format_stamp: "Formatted at 2026-09-27 21:30:07 on dist-test-slave-bxm0"
server_key: "55d24bccfd8ecfcbe8cc905254bc434b"
server_key_iv: "fc1e842703c8bf9739f43cfa3bae52e7"
server_key_version: "encryptionkey@0"
I20260927 21:30:07.457247 14363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:30:07.482151 14363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:30:07.486232 14363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:30:07.486557 14363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:30:07.487565 14363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:30:07.488924 14363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:30:07.489027 14363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:07.489178 14363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:30:07.489250 14363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:07.522330 14363 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45047
I20260927 21:30:07.522455 14490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45047 every 8 connection(s)
I20260927 21:30:07.523738 14363 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:30:07.532930 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 14363
I20260927 21:30:07.533146 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/wal/instance
I20260927 21:30:07.534057 388 external_mini_cluster.cc:1428] Setting key 7ff861e6d7a4e5e1c2e6ba787e966961
Sep 27 21:30:07 dist-test-slave-bxm0 krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:30:07.545023 14493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:07.526290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43587 (local address 127.0.97.62:37797)
0927 21:30:07.526745 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:07.526750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:07.527567 (+ 817us) server_negotiation.cc:407] Connection header received
0927 21:30:07.529374 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:07.529380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:07.529502 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:07.529618 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:07.531189 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:07.531936 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:07.534246 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:07.534718 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:07.534867 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:07.538981 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:07.538986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:07.538989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:07.539021 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:07.541928 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:07.542697 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:07.542701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:07.542704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:07.542833 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:07.543275 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:07.543284 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:07.543286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:07.543548 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:07.543700 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:07.544405 (+ 705us) server_negotiation.cc:299] Negotiation successful
0927 21:30:07.544631 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":246,"threads_started":1}
I20260927 21:30:07.546094 14494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:07.526724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.1:43587)
0927 21:30:07.527413 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:07.527449 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:07.529092 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:07.529894 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:07.529902 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:07.530401 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:30:07.531016 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:07.531032 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:07.532101 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:07.532111 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:07.534083 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:07.534099 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:07.534994 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:07.535003 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:07.535270 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:07.536032 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:30:07.536063 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:30:07.538742 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:30:07.542076 (+ 3334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:07.542086 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:07.542104 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:07.542555 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:07.542950 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:07.542955 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:07.542958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:07.543154 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:07.543677 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:30:07.543686 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:30:07.544031 (+ 345us) client_negotiation.cc:769] Sending connection context
0927 21:30:07.544298 (+ 267us) client_negotiation.cc:240] Negotiation successful
0927 21:30:07.544851 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":274,"threads_started":1}
I20260927 21:30:07.547763 14491 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37797
I20260927 21:30:07.548115 14491 heartbeater.cc:464] Registering TS with master...
I20260927 21:30:07.549077 14491 heartbeater.cc:511] Master 127.0.97.62:37797 requested a full tablet report, sending...
I20260927 21:30:07.551535 14302 ts_manager.cc:194] Registered new tserver with Master: f92ecd52d3014aeabccda938a17e7e68 (127.0.97.1:45047)
I20260927 21:30:07.553841 14302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43587
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 21:30:07.588292 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37797
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:30:07.927266 14497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:30:07.927573 14497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:30:07.927668 14497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:30:07.935416 14497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:30:07.935549 14497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:30:07.935664 14497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:30:07.945663 14497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:30:07.946923 14497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:30:07.948253 14497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:30:07.962033 14502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:07.961980 14503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:07.961998 14505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:30:07.963137 14497 server_base.cc:1034] running on GCE node
I20260927 21:30:07.963762 14497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:30:07.964721 14497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:30:07.965945 14497 hybrid_clock.cc:648] HybridClock initialized: now 1790544607965896 us; error 42 us; skew 500 ppm
I20260927 21:30:07.966260 14497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:30:07 dist-test-slave-bxm0 krb5kdc[14257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:30:07.969341 14497 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 21:30:07.970768 14497 webserver.cc:460] Webserver started at http://127.0.97.2:42471/ using document root <none> and password file <none>
I20260927 21:30:07.971380 14497 fs_manager.cc:359] Metadata directory not provided
I20260927 21:30:07.971487 14497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:30:07.971724 14497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:30:07.974625 14497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/data/instance:
uuid: "bfe26c8bab6e40c49bcde52dca387731"
format_stamp: "Formatted at 2026-09-27 21:30:07 on dist-test-slave-bxm0"
server_key: "da7eed9b93593dc5d02b8ec08e52737c"
server_key_iv: "4f6f66f7340b738d223977c1d20231f3"
server_key_version: "encryptionkey@0"
I20260927 21:30:07.975324 14497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bfe26c8bab6e40c49bcde52dca387731"
format_stamp: "Formatted at 2026-09-27 21:30:07 on dist-test-slave-bxm0"
server_key: "da7eed9b93593dc5d02b8ec08e52737c"
server_key_iv: "4f6f66f7340b738d223977c1d20231f3"
server_key_version: "encryptionkey@0"
I20260927 21:30:07.980298 14497 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:30:07.983628 14512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:07.984925 14497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 21:30:07.985087 14497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "bfe26c8bab6e40c49bcde52dca387731"
format_stamp: "Formatted at 2026-09-27 21:30:07 on dist-test-slave-bxm0"
server_key: "da7eed9b93593dc5d02b8ec08e52737c"
server_key_iv: "4f6f66f7340b738d223977c1d20231f3"
server_key_version: "encryptionkey@0"
I20260927 21:30:07.985286 14497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:30:08.019462 14497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:30:08.024139 14497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:30:08.024475 14497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:30:08.025568 14497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:30:08.027164 14497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:30:08.027242 14497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:08.027388 14497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:30:08.027441 14497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:08.061621 14497 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41425
I20260927 21:30:08.061761 14624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41425 every 8 connection(s)
I20260927 21:30:08.063125 14497 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:30:08.071890 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 14497
I20260927 21:30:08.072098 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/wal/instance
I20260927 21:30:08.072674 388 external_mini_cluster.cc:1428] Setting key f054c7b1b97317effa01a4eaa4785956
Sep 27 21:30:08 dist-test-slave-bxm0 krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:30:08.085059 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:08.065904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37057 (local address 127.0.97.62:37797)
0927 21:30:08.066317 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:08.066322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:08.067396 (+ 1074us) server_negotiation.cc:407] Connection header received
0927 21:30:08.069156 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:08.069161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:08.069232 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:08.069333 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:08.070923 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.071616 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.073965 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.074587 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.074744 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:08.079120 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:08.079125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:08.079129 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:08.079162 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:08.081929 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:08.082716 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:08.082720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:08.082722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:08.082845 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:08.083284 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:08.083289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:08.083291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:08.083558 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:08.083714 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:08.084424 (+ 710us) server_negotiation.cc:299] Negotiation successful
0927 21:30:08.084662 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":194,"threads_started":1}
I20260927 21:30:08.086109 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:08.066560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.2:37057)
0927 21:30:08.067231 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:08.067272 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:08.068890 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:08.069553 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:08.069561 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:08.070159 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:30:08.070732 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:08.070749 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.071775 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.071781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:08.073782 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:08.073799 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.074835 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.074840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:08.075162 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:08.075930 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:30:08.075969 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:30:08.078844 (+ 2875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:30:08.082091 (+ 3247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:08.082100 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:08.082119 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:08.082571 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:08.082963 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:08.082968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:08.082972 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:08.083158 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:08.083689 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:30:08.083697 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:30:08.084059 (+ 362us) client_negotiation.cc:769] Sending connection context
0927 21:30:08.084322 (+ 263us) client_negotiation.cc:240] Negotiation successful
0927 21:30:08.084872 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":279,"threads_started":1}
I20260927 21:30:08.087607 14625 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37797
I20260927 21:30:08.087942 14625 heartbeater.cc:464] Registering TS with master...
I20260927 21:30:08.088893 14625 heartbeater.cc:511] Master 127.0.97.62:37797 requested a full tablet report, sending...
I20260927 21:30:08.090638 14302 ts_manager.cc:194] Registered new tserver with Master: bfe26c8bab6e40c49bcde52dca387731 (127.0.97.2:41425)
I20260927 21:30:08.091945 14302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37057
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 21:30:08.124954 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37797
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:30:08.453176 14631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:30:08.453466 14631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:30:08.453672 14631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:30:08.461104 14631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:30:08.461248 14631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:30:08.461445 14631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:30:08.474987 14631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:30:08.476402 14631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:30:08.477797 14631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:30:08.491343 14636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:08.491398 14639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:08.491398 14637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:30:08.492313 14631 server_base.cc:1034] running on GCE node
I20260927 21:30:08.492947 14631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:30:08.494015 14631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:30:08.495286 14631 hybrid_clock.cc:648] HybridClock initialized: now 1790544608495225 us; error 40 us; skew 500 ppm
I20260927 21:30:08.495676 14631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:30:08 dist-test-slave-bxm0 krb5kdc[14257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:30:08.499482 14631 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 21:30:08.501052 14631 webserver.cc:460] Webserver started at http://127.0.97.3:44509/ using document root <none> and password file <none>
I20260927 21:30:08.501924 14631 fs_manager.cc:359] Metadata directory not provided
I20260927 21:30:08.502044 14631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:30:08.502326 14631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:30:08.505589 14631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/data/instance:
uuid: "b0ae63d02c9345f996a817c27132fb72"
format_stamp: "Formatted at 2026-09-27 21:30:08 on dist-test-slave-bxm0"
server_key: "8842a7693551bcd9dabad6498a38885a"
server_key_iv: "b2e8fa57a55481976f477e6953432d67"
server_key_version: "encryptionkey@0"
I20260927 21:30:08.506600 14631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b0ae63d02c9345f996a817c27132fb72"
format_stamp: "Formatted at 2026-09-27 21:30:08 on dist-test-slave-bxm0"
server_key: "8842a7693551bcd9dabad6498a38885a"
server_key_iv: "b2e8fa57a55481976f477e6953432d67"
server_key_version: "encryptionkey@0"
I20260927 21:30:08.512248 14631 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 21:30:08.516027 14646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:08.517446 14631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 21:30:08.517678 14631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "b0ae63d02c9345f996a817c27132fb72"
format_stamp: "Formatted at 2026-09-27 21:30:08 on dist-test-slave-bxm0"
server_key: "8842a7693551bcd9dabad6498a38885a"
server_key_iv: "b2e8fa57a55481976f477e6953432d67"
server_key_version: "encryptionkey@0"
I20260927 21:30:08.517868 14631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:30:08.540136 14631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:30:08.545100 14631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:30:08.545423 14631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:30:08.546518 14631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:30:08.547981 14631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:30:08.548062 14631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:08.548249 14631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:30:08.548300 14631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:08.557886 14491 heartbeater.cc:503] Master 127.0.97.62:37797 was elected leader, sending a full tablet report...
I20260927 21:30:08.582120 14631 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41909
I20260927 21:30:08.582168 14758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41909 every 8 connection(s)
I20260927 21:30:08.583416 14631 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:30:08.587802 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 14631
I20260927 21:30:08.587971 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/wal/instance
I20260927 21:30:08.588452 388 external_mini_cluster.cc:1428] Setting key a2688d431f7b96f3f090fc63a012a270
Sep 27 21:30:08 dist-test-slave-bxm0 krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:30:08.603914 14761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:08.586141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48531 (local address 127.0.97.62:37797)
0927 21:30:08.586540 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:08.586544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:08.587717 (+ 1173us) server_negotiation.cc:407] Connection header received
0927 21:30:08.589807 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:08.589812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:08.589922 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:08.590045 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:08.591515 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.592314 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.593947 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.594330 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.594466 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:08.598310 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:08.598315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:08.598318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:08.598353 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:08.600725 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:08.601471 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:08.601478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:08.601486 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:08.601601 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:08.602098 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:08.602102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:08.602107 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:08.602384 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:08.602545 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:08.603276 (+ 731us) server_negotiation.cc:299] Negotiation successful
0927 21:30:08.603517 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":203,"threads_started":1}
I20260927 21:30:08.604892 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:08.586784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.3:48531)
0927 21:30:08.587562 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:08.587599 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:08.589493 (+ 1894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:08.590279 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:08.590287 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:08.590806 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:30:08.591345 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:08.591372 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.592474 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.592485 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:08.593790 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:08.593802 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.594562 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.594569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:08.594924 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:08.595772 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:30:08.595800 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:30:08.598074 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:30:08.600879 (+ 2805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:08.600889 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:08.600907 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:08.601332 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:08.601759 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:08.601764 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:08.601768 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:08.601976 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:08.602529 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:30:08.602537 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:30:08.602889 (+ 352us) client_negotiation.cc:769] Sending connection context
0927 21:30:08.603166 (+ 277us) client_negotiation.cc:240] Negotiation successful
0927 21:30:08.603742 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":326,"threads_started":1}
I20260927 21:30:08.606253 14759 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37797
I20260927 21:30:08.606523 14759 heartbeater.cc:464] Registering TS with master...
I20260927 21:30:08.607199 14759 heartbeater.cc:511] Master 127.0.97.62:37797 requested a full tablet report, sending...
I20260927 21:30:08.608562 14302 ts_manager.cc:194] Registered new tserver with Master: b0ae63d02c9345f996a817c27132fb72 (127.0.97.3:41909)
I20260927 21:30:08.609388 14302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48531
I20260927 21:30:08.615971 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:30:08.634533 14761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:08.620498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53940 (local address 127.0.97.62:37797)
0927 21:30:08.620662 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:08.620666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:08.621190 (+ 524us) server_negotiation.cc:407] Connection header received
0927 21:30:08.621366 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:08.621390 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:08.621455 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:08.621541 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:08.623096 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.623800 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.625829 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.626341 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.626452 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:08.629050 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:08.629055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:08.629058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:08.629092 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:08.631985 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:08.632697 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:08.632701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:08.632704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:08.632823 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:08.633149 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:08.633152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:08.633154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:08.633482 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:08.633751 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:08.634109 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 21:30:08.634290 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:30:08.638792 14302 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:30:08.646013 14302 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:30:08.671108 14774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:08.659849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41425 (local address 127.0.0.1:48788)
0927 21:30:08.660791 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:08.660815 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:08.660972 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:08.662890 (+ 1918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:08.662898 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:08.662928 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:30:08.663649 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:08.663665 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.665872 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.665877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:08.668430 (+ 2553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:08.668445 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.670282 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.670288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:08.670511 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:08.670556 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 21:30:08.670666 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 21:30:08.670801 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":172,"thread_start_us":854,"threads_started":1}
I20260927 21:30:08.671141 14776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:08.660536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48788 (local address 127.0.97.2:41425)
0927 21:30:08.662178 (+ 1642us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:08.662189 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:08.662226 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:30:08.662338 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:08.662347 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:08.662633 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:08.663406 (+ 773us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:30:08.664005 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.665709 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.668586 (+ 2877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.670047 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.670394 (+ 347us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:08.670532 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:08.670779 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 21:30:08.670917 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1472,"thread_start_us":1362,"threads_started":1}
I20260927 21:30:08.672093 14772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:08.658986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45047 (local address 127.0.0.1:37460)
0927 21:30:08.659709 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:08.659756 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:08.659941 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:08.663273 (+ 3332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:08.663277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:08.663294 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:30:08.664007 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:08.664024 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.665576 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.665581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:08.668352 (+ 2771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:08.668367 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.670229 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.670235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:08.671439 (+ 1204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:08.671473 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 21:30:08.671576 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 21:30:08.671703 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":977,"thread_start_us":301,"threads_started":1}
I20260927 21:30:08.672118 14773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:08.659319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37460 (local address 127.0.97.1:45047)
0927 21:30:08.662682 (+ 3363us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:08.662692 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:08.662730 (+ 38us) server_negotiation.cc:407] Connection header received
0927 21:30:08.662838 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:08.662845 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:08.663049 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:08.663829 (+ 780us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:30:08.664200 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.665417 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.668555 (+ 3138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.670013 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.670160 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:08.670384 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:08.671630 (+ 1246us) server_negotiation.cc:299] Negotiation successful
0927 21:30:08.671784 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3183,"thread_start_us":3172,"threads_started":1}
I20260927 21:30:08.673538 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:08.660158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41909 (local address 127.0.0.1:46560)
0927 21:30:08.661540 (+ 1382us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:08.661556 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:08.661673 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:08.663238 (+ 1565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:08.663246 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:08.663275 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:30:08.664127 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:08.664146 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.666071 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.666079 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:08.668026 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:08.668039 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.670164 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.670174 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:08.672847 (+ 2673us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:08.672880 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 21:30:08.672986 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 21:30:08.673116 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1277,"mutex_wait_us":2474,"thread_start_us":1148,"threads_started":1}
I20260927 21:30:08.674293 14777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:08.660550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46560 (local address 127.0.97.3:41909)
0927 21:30:08.662639 (+ 2089us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:08.662647 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:08.662679 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:30:08.662771 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:08.662778 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:08.663005 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:08.663208 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:30:08.664336 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.665900 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.668193 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.669598 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.669771 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:08.670016 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:08.673781 (+ 3765us) server_negotiation.cc:299] Negotiation successful
0927 21:30:08.673967 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1931,"thread_start_us":1770,"threads_started":1}
I20260927 21:30:08.675488 14560 tablet_service.cc:1467] Processing CreateTablet for tablet f68f9655d4ef452cb738370abfb2d4c2 (DEFAULT_TABLE table=test-table [id=5272d322387f4665bf8ff1169e9c4dc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:30:08.677011 14560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68f9655d4ef452cb738370abfb2d4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:30:08.677486 14694 tablet_service.cc:1467] Processing CreateTablet for tablet f68f9655d4ef452cb738370abfb2d4c2 (DEFAULT_TABLE table=test-table [id=5272d322387f4665bf8ff1169e9c4dc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:30:08.677917 14426 tablet_service.cc:1467] Processing CreateTablet for tablet f68f9655d4ef452cb738370abfb2d4c2 (DEFAULT_TABLE table=test-table [id=5272d322387f4665bf8ff1169e9c4dc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:30:08.678859 14694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68f9655d4ef452cb738370abfb2d4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:30:08.679189 14426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68f9655d4ef452cb738370abfb2d4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:30:08.691629 14778 tablet_bootstrap.cc:492] T f68f9655d4ef452cb738370abfb2d4c2 P f92ecd52d3014aeabccda938a17e7e68: Bootstrap starting.
I20260927 21:30:08.693045 14779 tablet_bootstrap.cc:492] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72: Bootstrap starting.
I20260927 21:30:08.695086 14780 tablet_bootstrap.cc:492] T f68f9655d4ef452cb738370abfb2d4c2 P bfe26c8bab6e40c49bcde52dca387731: Bootstrap starting.
I20260927 21:30:08.695868 14778 tablet_bootstrap.cc:654] T f68f9655d4ef452cb738370abfb2d4c2 P f92ecd52d3014aeabccda938a17e7e68: Neither blocks nor log segments found. Creating new log.
I20260927 21:30:08.696455 14779 tablet_bootstrap.cc:654] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72: Neither blocks nor log segments found. Creating new log.
I20260927 21:30:08.697184 14778 log.cc:824] T f68f9655d4ef452cb738370abfb2d4c2 P f92ecd52d3014aeabccda938a17e7e68: Log is configured to *not* fsync() on all Append() calls
I20260927 21:30:08.697657 14779 log.cc:824] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72: Log is configured to *not* fsync() on all Append() calls
I20260927 21:30:08.698161 14780 tablet_bootstrap.cc:654] T f68f9655d4ef452cb738370abfb2d4c2 P bfe26c8bab6e40c49bcde52dca387731: Neither blocks nor log segments found. Creating new log.
I20260927 21:30:08.699363 14780 log.cc:824] T f68f9655d4ef452cb738370abfb2d4c2 P bfe26c8bab6e40c49bcde52dca387731: Log is configured to *not* fsync() on all Append() calls
I20260927 21:30:08.700482 14778 tablet_bootstrap.cc:492] T f68f9655d4ef452cb738370abfb2d4c2 P f92ecd52d3014aeabccda938a17e7e68: No bootstrap required, opened a new log
I20260927 21:30:08.700698 14779 tablet_bootstrap.cc:492] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72: No bootstrap required, opened a new log
I20260927 21:30:08.700773 14778 ts_tablet_manager.cc:1397] T f68f9655d4ef452cb738370abfb2d4c2 P f92ecd52d3014aeabccda938a17e7e68: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 21:30:08.700986 14779 ts_tablet_manager.cc:1397] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260927 21:30:08.702231 14780 tablet_bootstrap.cc:492] T f68f9655d4ef452cb738370abfb2d4c2 P bfe26c8bab6e40c49bcde52dca387731: No bootstrap required, opened a new log
I20260927 21:30:08.702765 14780 ts_tablet_manager.cc:1397] T f68f9655d4ef452cb738370abfb2d4c2 P bfe26c8bab6e40c49bcde52dca387731: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 21:30:08.708777 14779 raft_consensus.cc:348] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 } } peers { permanent_uuid: "b0ae63d02c9345f996a817c27132fb72" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41909 } } peers { permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } }
I20260927 21:30:08.709251 14779 raft_consensus.cc:374] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:30:08.709398 14779 raft_consensus.cc:729] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ae63d02c9345f996a817c27132fb72, State: Initialized, Role: FOLLOWER
I20260927 21:30:08.710152 14779 consensus_queue.cc:260] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 } } peers { permanent_uuid: "b0ae63d02c9345f996a817c27132fb72" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41909 } } peers { permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } }
I20260927 21:30:08.711333 14759 heartbeater.cc:503] Master 127.0.97.62:37797 was elected leader, sending a full tablet report...
I20260927 21:30:08.712126 14779 ts_tablet_manager.cc:1428] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72: Time spent starting tablet: real 0.011s user 0.003s sys 0.008s
I20260927 21:30:08.711809 14778 raft_consensus.cc:348] T f68f9655d4ef452cb738370abfb2d4c2 P f92ecd52d3014aeabccda938a17e7e68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 } } peers { permanent_uuid: "b0ae63d02c9345f996a817c27132fb72" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41909 } } peers { permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } }
I20260927 21:30:08.712301 14778 raft_consensus.cc:374] T f68f9655d4ef452cb738370abfb2d4c2 P f92ecd52d3014aeabccda938a17e7e68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:30:08.712396 14778 raft_consensus.cc:729] T f68f9655d4ef452cb738370abfb2d4c2 P f92ecd52d3014aeabccda938a17e7e68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f92ecd52d3014aeabccda938a17e7e68, State: Initialized, Role: FOLLOWER
I20260927 21:30:08.713039 14778 consensus_queue.cc:260] T f68f9655d4ef452cb738370abfb2d4c2 P f92ecd52d3014aeabccda938a17e7e68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 } } peers { permanent_uuid: "b0ae63d02c9345f996a817c27132fb72" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41909 } } peers { permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } }
I20260927 21:30:08.713558 14780 raft_consensus.cc:348] T f68f9655d4ef452cb738370abfb2d4c2 P bfe26c8bab6e40c49bcde52dca387731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 } } peers { permanent_uuid: "b0ae63d02c9345f996a817c27132fb72" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41909 } } peers { permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } }
I20260927 21:30:08.714120 14780 raft_consensus.cc:374] T f68f9655d4ef452cb738370abfb2d4c2 P bfe26c8bab6e40c49bcde52dca387731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:30:08.714253 14780 raft_consensus.cc:729] T f68f9655d4ef452cb738370abfb2d4c2 P bfe26c8bab6e40c49bcde52dca387731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfe26c8bab6e40c49bcde52dca387731, State: Initialized, Role: FOLLOWER
I20260927 21:30:08.715011 14778 ts_tablet_manager.cc:1428] T f68f9655d4ef452cb738370abfb2d4c2 P f92ecd52d3014aeabccda938a17e7e68: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260927 21:30:08.715020 14780 consensus_queue.cc:260] T f68f9655d4ef452cb738370abfb2d4c2 P bfe26c8bab6e40c49bcde52dca387731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 } } peers { permanent_uuid: "b0ae63d02c9345f996a817c27132fb72" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41909 } } peers { permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } }
I20260927 21:30:08.716917 14780 ts_tablet_manager.cc:1428] T f68f9655d4ef452cb738370abfb2d4c2 P bfe26c8bab6e40c49bcde52dca387731: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260927 21:30:08.717120 14625 heartbeater.cc:503] Master 127.0.97.62:37797 was elected leader, sending a full tablet report...
W20260927 21:30:08.776201 14492 tablet.cc:2369] T f68f9655d4ef452cb738370abfb2d4c2 P f92ecd52d3014aeabccda938a17e7e68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:30:08.796770 14784 raft_consensus.cc:482] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:30:08.797050 14784 raft_consensus.cc:504] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 } } peers { permanent_uuid: "b0ae63d02c9345f996a817c27132fb72" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41909 } } peers { permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } }
I20260927 21:30:08.798748 14784 leader_election.cc:288] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f92ecd52d3014aeabccda938a17e7e68 (127.0.97.1:45047), bfe26c8bab6e40c49bcde52dca387731 (127.0.97.2:41425)
I20260927 21:30:08.805593 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:08.799070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45047 (local address 127.0.97.3:58547)
0927 21:30:08.799446 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:08.799469 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:08.799630 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:08.800044 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:08.800049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:08.800096 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:30:08.800285 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:08.800293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.801653 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.801661 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:08.803576 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:08.803590 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.804832 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.804839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:08.805049 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:08.805094 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 21:30:08.805185 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 21:30:08.805336 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":83,"spinlock_wait_cycles":34176}
I20260927 21:30:08.805640 14773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:08.799191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58547 (local address 127.0.97.1:45047)
0927 21:30:08.799576 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:08.799586 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:08.799617 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:30:08.799816 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:08.799824 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:08.799907 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:08.799991 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:30:08.800433 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.801448 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.803733 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.804667 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.804783 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:08.804863 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:08.805229 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 21:30:08.805364 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260927 21:30:08.806035 14776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:08.799265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50295 (local address 127.0.97.2:41425)
0927 21:30:08.799448 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:08.799455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:08.800085 (+ 630us) server_negotiation.cc:407] Connection header received
0927 21:30:08.800225 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:08.800229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:08.800293 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:08.800395 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:30:08.800843 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.801885 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.804117 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.805039 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.805184 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:08.805275 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:08.805654 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 21:30:08.805821 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 21:30:08.806102 14787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:08.799070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41425 (local address 127.0.97.3:50295)
0927 21:30:08.799939 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:08.799952 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:08.800091 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:08.800409 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:08.800416 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:08.800445 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:30:08.800689 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:08.800694 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.802091 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.802098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:08.803956 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:08.803971 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.805282 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.805288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:08.805457 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:08.805489 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 21:30:08.805586 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 21:30:08.805757 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":139,"thread_start_us":457,"threads_started":1}
I20260927 21:30:08.807051 14446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68f9655d4ef452cb738370abfb2d4c2" candidate_uuid: "b0ae63d02c9345f996a817c27132fb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f92ecd52d3014aeabccda938a17e7e68" is_pre_election: true
I20260927 21:30:08.807065 14580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68f9655d4ef452cb738370abfb2d4c2" candidate_uuid: "b0ae63d02c9345f996a817c27132fb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe26c8bab6e40c49bcde52dca387731" is_pre_election: true
I20260927 21:30:08.807554 14580 raft_consensus.cc:2454] T f68f9655d4ef452cb738370abfb2d4c2 P bfe26c8bab6e40c49bcde52dca387731 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0ae63d02c9345f996a817c27132fb72 in term 0.
I20260927 21:30:08.807554 14446 raft_consensus.cc:2454] T f68f9655d4ef452cb738370abfb2d4c2 P f92ecd52d3014aeabccda938a17e7e68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0ae63d02c9345f996a817c27132fb72 in term 0.
I20260927 21:30:08.808259 14648 leader_election.cc:302] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b0ae63d02c9345f996a817c27132fb72, bfe26c8bab6e40c49bcde52dca387731, f92ecd52d3014aeabccda938a17e7e68; no voters:
I20260927 21:30:08.808674 14784 raft_consensus.cc:2781] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:30:08.808837 14784 raft_consensus.cc:482] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:30:08.808957 14784 raft_consensus.cc:3037] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:30:08.811333 14784 raft_consensus.cc:504] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 } } peers { permanent_uuid: "b0ae63d02c9345f996a817c27132fb72" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41909 } } peers { permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } }
I20260927 21:30:08.811977 14784 leader_election.cc:288] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72 [CANDIDATE]: Term 1 election: Requested vote from peers f92ecd52d3014aeabccda938a17e7e68 (127.0.97.1:45047), bfe26c8bab6e40c49bcde52dca387731 (127.0.97.2:41425)
I20260927 21:30:08.812561 14446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68f9655d4ef452cb738370abfb2d4c2" candidate_uuid: "b0ae63d02c9345f996a817c27132fb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f92ecd52d3014aeabccda938a17e7e68"
I20260927 21:30:08.812562 14580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68f9655d4ef452cb738370abfb2d4c2" candidate_uuid: "b0ae63d02c9345f996a817c27132fb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe26c8bab6e40c49bcde52dca387731"
I20260927 21:30:08.812785 14446 raft_consensus.cc:3037] T f68f9655d4ef452cb738370abfb2d4c2 P f92ecd52d3014aeabccda938a17e7e68 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:30:08.812803 14580 raft_consensus.cc:3037] T f68f9655d4ef452cb738370abfb2d4c2 P bfe26c8bab6e40c49bcde52dca387731 [term 0 FOLLOWER]: Advancing to term 1
W20260927 21:30:08.815231 14626 tablet.cc:2369] T f68f9655d4ef452cb738370abfb2d4c2 P bfe26c8bab6e40c49bcde52dca387731: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:30:08.816123 14580 raft_consensus.cc:2454] T f68f9655d4ef452cb738370abfb2d4c2 P bfe26c8bab6e40c49bcde52dca387731 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0ae63d02c9345f996a817c27132fb72 in term 1.
I20260927 21:30:08.816121 14446 raft_consensus.cc:2454] T f68f9655d4ef452cb738370abfb2d4c2 P f92ecd52d3014aeabccda938a17e7e68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0ae63d02c9345f996a817c27132fb72 in term 1.
I20260927 21:30:08.816627 14648 leader_election.cc:302] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b0ae63d02c9345f996a817c27132fb72, bfe26c8bab6e40c49bcde52dca387731, f92ecd52d3014aeabccda938a17e7e68; no voters:
I20260927 21:30:08.816982 14784 raft_consensus.cc:2781] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:30:08.817462 14784 raft_consensus.cc:686] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72 [term 1 LEADER]: Becoming Leader. State: Replica: b0ae63d02c9345f996a817c27132fb72, State: Running, Role: LEADER
I20260927 21:30:08.817952 14784 consensus_queue.cc:237] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 } } peers { permanent_uuid: "b0ae63d02c9345f996a817c27132fb72" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41909 } } peers { permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } }
I20260927 21:30:08.822943 14301 catalog_manager.cc:5495] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0ae63d02c9345f996a817c27132fb72 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "b0ae63d02c9345f996a817c27132fb72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0ae63d02c9345f996a817c27132fb72" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } health_report { overall_health: UNKNOWN } } }
W20260927 21:30:08.835086 14760 tablet.cc:2369] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:30:08.870011 14777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:08.864942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46564 (local address 127.0.97.3:41909)
0927 21:30:08.865119 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:08.865124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:08.865141 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:30:08.865320 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:08.865323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:08.865415 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:08.865545 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:30:08.865923 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.866624 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.868006 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.868422 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.868532 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:08.868966 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:30:08.869485 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:30:08.869596 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:08.869732 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 21:30:08.869804 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 21:30:08.878540 14446 raft_consensus.cc:1261] T f68f9655d4ef452cb738370abfb2d4c2 P f92ecd52d3014aeabccda938a17e7e68 [term 1 FOLLOWER]: Refusing update from remote peer b0ae63d02c9345f996a817c27132fb72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:30:08.878542 14580 raft_consensus.cc:1261] T f68f9655d4ef452cb738370abfb2d4c2 P bfe26c8bab6e40c49bcde52dca387731 [term 1 FOLLOWER]: Refusing update from remote peer b0ae63d02c9345f996a817c27132fb72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:30:08.879573 14788 consensus_queue.cc:1035] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:30:08.879974 14784 consensus_queue.cc:1035] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:30:08.898257 14791 mvcc.cc:205] Tried to move back new op lower bound from 7334070717955694592 to 7334070717722759168. Current Snapshot: MvccSnapshot[applied={T|T < 7334070717955694592}]
I20260927 21:30:08.913375 14301 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:30:08.913713 14301 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:30:08.917532 14301 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751: Sending DeleteTablet for 3 replicas of tablet f68f9655d4ef452cb738370abfb2d4c2
I20260927 21:30:08.918955 14694 tablet_service.cc:1514] Processing DeleteTablet for tablet f68f9655d4ef452cb738370abfb2d4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:30:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46560
I20260927 21:30:08.919003 14426 tablet_service.cc:1514] Processing DeleteTablet for tablet f68f9655d4ef452cb738370abfb2d4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:30:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37460
I20260927 21:30:08.919051 14560 tablet_service.cc:1514] Processing DeleteTablet for tablet f68f9655d4ef452cb738370abfb2d4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:30:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48788
I20260927 21:30:08.920491 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 14363
I20260927 21:30:08.935457 14801 tablet_replica.cc:330] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72: stopping tablet replica
I20260927 21:30:08.936163 14801 raft_consensus.cc:2229] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:30:08.936998 14801 raft_consensus.cc:2258] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:30:08.939891 14801 ts_tablet_manager.cc:1905] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:30:08.942132 14803 tablet_replica.cc:330] T f68f9655d4ef452cb738370abfb2d4c2 P f92ecd52d3014aeabccda938a17e7e68: stopping tablet replica
I20260927 21:30:08.942554 14802 tablet_replica.cc:330] T f68f9655d4ef452cb738370abfb2d4c2 P bfe26c8bab6e40c49bcde52dca387731: stopping tablet replica
I20260927 21:30:08.942605 14803 raft_consensus.cc:2229] T f68f9655d4ef452cb738370abfb2d4c2 P f92ecd52d3014aeabccda938a17e7e68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:30:08.943058 14802 raft_consensus.cc:2229] T f68f9655d4ef452cb738370abfb2d4c2 P bfe26c8bab6e40c49bcde52dca387731 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:30:08.943297 14803 raft_consensus.cc:2258] T f68f9655d4ef452cb738370abfb2d4c2 P f92ecd52d3014aeabccda938a17e7e68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:30:08.943529 14802 raft_consensus.cc:2258] T f68f9655d4ef452cb738370abfb2d4c2 P bfe26c8bab6e40c49bcde52dca387731 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 21:30:08 dist-test-slave-bxm0 krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 21:30:08.945761 14803 ts_tablet_manager.cc:1905] T f68f9655d4ef452cb738370abfb2d4c2 P f92ecd52d3014aeabccda938a17e7e68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:30:08.950297 14802 ts_tablet_manager.cc:1905] T f68f9655d4ef452cb738370abfb2d4c2 P bfe26c8bab6e40c49bcde52dca387731: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:30:08.953699 14803 ts_tablet_manager.cc:1918] T f68f9655d4ef452cb738370abfb2d4c2 P f92ecd52d3014aeabccda938a17e7e68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:30:08.953967 14803 log.cc:1196] T f68f9655d4ef452cb738370abfb2d4c2 P f92ecd52d3014aeabccda938a17e7e68: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/wal/wals/f68f9655d4ef452cb738370abfb2d4c2
I20260927 21:30:08.954617 14803 ts_tablet_manager.cc:1939] T f68f9655d4ef452cb738370abfb2d4c2 P f92ecd52d3014aeabccda938a17e7e68: Deleting consensus metadata
I20260927 21:30:08.954691 14773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:08.921927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37462 (local address 127.0.97.1:45047)
0927 21:30:08.922145 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:08.922155 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:08.927791 (+ 5636us) server_negotiation.cc:407] Connection header received
0927 21:30:08.928897 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:08.928906 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:08.929062 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:08.929158 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:08.934525 (+ 5367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.935834 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.939839 (+ 4005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:08.940514 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:08.940644 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:08.946782 (+ 6138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:08.946798 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:08.946815 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:08.946864 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:08.950895 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:08.951866 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:08.951875 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:08.951880 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:08.952075 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:08.952597 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:08.952602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:08.952606 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:08.953017 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:08.953300 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:08.953970 (+ 670us) server_negotiation.cc:299] Negotiation successful
0927 21:30:08.954244 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:30:08.955952 14486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:30:08.957283 14802 ts_tablet_manager.cc:1918] T f68f9655d4ef452cb738370abfb2d4c2 P bfe26c8bab6e40c49bcde52dca387731: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:30:08.957460 14802 log.cc:1196] T f68f9655d4ef452cb738370abfb2d4c2 P bfe26c8bab6e40c49bcde52dca387731: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/wal/wals/f68f9655d4ef452cb738370abfb2d4c2
I20260927 21:30:08.958039 14802 ts_tablet_manager.cc:1939] T f68f9655d4ef452cb738370abfb2d4c2 P bfe26c8bab6e40c49bcde52dca387731: Deleting consensus metadata
I20260927 21:30:08.960067 14287 catalog_manager.cc:4841] TS bfe26c8bab6e40c49bcde52dca387731 (127.0.97.2:41425): tablet f68f9655d4ef452cb738370abfb2d4c2 (table test-table [id=5272d322387f4665bf8ff1169e9c4dc6]) successfully deleted
I20260927 21:30:08.962050 14801 ts_tablet_manager.cc:1918] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:30:08.962240 14801 log.cc:1196] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/wal/wals/f68f9655d4ef452cb738370abfb2d4c2
I20260927 21:30:08.962836 14801 ts_tablet_manager.cc:1939] T f68f9655d4ef452cb738370abfb2d4c2 P b0ae63d02c9345f996a817c27132fb72: Deleting consensus metadata
I20260927 21:30:08.964771 14288 catalog_manager.cc:4841] TS b0ae63d02c9345f996a817c27132fb72 (127.0.97.3:41909): tablet f68f9655d4ef452cb738370abfb2d4c2 (table test-table [id=5272d322387f4665bf8ff1169e9c4dc6]) successfully deleted
I20260927 21:30:09.070863 14288 catalog_manager.cc:4841] TS f92ecd52d3014aeabccda938a17e7e68 (127.0.97.1:45047): tablet f68f9655d4ef452cb738370abfb2d4c2 (table test-table [id=5272d322387f4665bf8ff1169e9c4dc6]) successfully deleted
I20260927 21:30:09.071491 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 14363
I20260927 21:30:09.090515 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 14497
Sep 27 21:30:09 dist-test-slave-bxm0 krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 21:30:09.107685 14776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:09.091509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48792 (local address 127.0.97.2:41425)
0927 21:30:09.091682 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:09.091687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:09.091706 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:30:09.091760 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:09.091763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:09.091837 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:09.091922 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:09.093665 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:09.094447 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:09.096347 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:09.096723 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:09.096814 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:09.101956 (+ 5142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:09.101970 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:09.101986 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:09.102024 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:09.104854 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:09.105539 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:09.105547 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:09.105549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:09.105740 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:09.106108 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:09.106113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:09.106117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:09.106493 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:09.106783 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:09.107101 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 21:30:09.107324 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 21:30:09.108312 14620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:30:09.200663 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 14497
I20260927 21:30:09.218003 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 14631
Sep 27 21:30:09 dist-test-slave-bxm0 krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 21:30:09.235173 14777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:09.218794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46568 (local address 127.0.97.3:41909)
0927 21:30:09.219002 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:09.219009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:09.219028 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:30:09.219235 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:09.219241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:09.219315 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:09.219431 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:09.220892 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:09.222031 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:09.223446 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:09.223896 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:09.224050 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:09.228691 (+ 4641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:09.228703 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:09.228721 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:09.228769 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:09.232154 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:09.232879 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:09.232889 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:09.232892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:09.233061 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:09.233451 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:09.233457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:09.233460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:09.233882 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:09.234255 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:09.234544 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 21:30:09.234790 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 21:30:09.235847 14754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:30:09.357296 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 14631
I20260927 21:30:09.377126 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 14270
I20260927 21:30:09.378355 14332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:30:09.502749 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 14270
I20260927 21:30:09.524302 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37797
--webserver_interface=127.0.97.62
--webserver_port=40891
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:30:09.861899 14809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:30:09.862288 14809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:30:09.862449 14809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:30:09.870621 14809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:30:09.870786 14809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:30:09.870898 14809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:30:09.871001 14809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:30:09.871088 14809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:30:09.881387 14809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=40891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:30:09.882833 14809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:30:09.884205 14809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:30:09.907287 14818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:09.907271 14816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:09.907634 14815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:30:09.908244 14809 server_base.cc:1034] running on GCE node
I20260927 21:30:09.909191 14809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:30:09.910876 14809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:30:09.912132 14809 hybrid_clock.cc:648] HybridClock initialized: now 1790544609912079 us; error 46 us; skew 500 ppm
I20260927 21:30:09.912602 14809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:30:09 dist-test-slave-bxm0 krb5kdc[14257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:30:09.920228 14809 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:30:09.922143 14809 webserver.cc:460] Webserver started at http://127.0.97.62:40891/ using document root <none> and password file <none>
I20260927 21:30:09.922966 14809 fs_manager.cc:359] Metadata directory not provided
I20260927 21:30:09.923141 14809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:30:09.929723 14809 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.004s
I20260927 21:30:09.932833 14825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:09.934367 14809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 21:30:09.934584 14809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "76f94b335cdc416d9b1742c8e36dc751"
format_stamp: "Formatted at 2026-09-27 21:30:06 on dist-test-slave-bxm0"
server_key: "7f05210ad6401a58ef0ee35f900a0a14"
server_key_iv: "677e47646ef496327190ebca3c0fb114"
server_key_version: "encryptionkey@0"
I20260927 21:30:09.935762 14809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:30:09.951510 14809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:30:09.955890 14809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:30:09.956205 14809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:30:09.975893 14809 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37797
I20260927 21:30:09.975926 14876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37797 every 8 connection(s)
I20260927 21:30:09.977519 14809 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:30:09.984603 14877 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:30:09.987933 14877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751: Bootstrap starting.
I20260927 21:30:09.988247 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 14809
I20260927 21:30:09.989007 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45047
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37797
--webserver_port=42881
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:30:10.007886 14877 log.cc:824] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751: Log is configured to *not* fsync() on all Append() calls
I20260927 21:30:10.020154 14877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:30:10.020725 14877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751: Bootstrap complete.
I20260927 21:30:10.030190 14877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f94b335cdc416d9b1742c8e36dc751" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37797 } }
I20260927 21:30:10.031064 14877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f94b335cdc416d9b1742c8e36dc751, State: Initialized, Role: FOLLOWER
I20260927 21:30:10.031777 14877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f94b335cdc416d9b1742c8e36dc751" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37797 } }
I20260927 21:30:10.032049 14877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:30:10.032198 14877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:30:10.032388 14877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:30:10.036361 14877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f94b335cdc416d9b1742c8e36dc751" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37797 } }
I20260927 21:30:10.036940 14877 leader_election.cc:302] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76f94b335cdc416d9b1742c8e36dc751; no voters:
I20260927 21:30:10.037632 14877 leader_election.cc:288] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 21:30:10.037786 14881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:30:10.039106 14881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [term 2 LEADER]: Becoming Leader. State: Replica: 76f94b335cdc416d9b1742c8e36dc751, State: Running, Role: LEADER
I20260927 21:30:10.039822 14881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f94b335cdc416d9b1742c8e36dc751" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37797 } }
I20260927 21:30:10.040241 14877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:30:10.042686 14883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76f94b335cdc416d9b1742c8e36dc751. Latest consensus state: current_term: 2 leader_uuid: "76f94b335cdc416d9b1742c8e36dc751" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f94b335cdc416d9b1742c8e36dc751" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37797 } } }
I20260927 21:30:10.043228 14883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [sys.catalog]: This master's current role is: LEADER
I20260927 21:30:10.043062 14882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "76f94b335cdc416d9b1742c8e36dc751" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f94b335cdc416d9b1742c8e36dc751" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37797 } } }
I20260927 21:30:10.043536 14882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [sys.catalog]: This master's current role is: LEADER
I20260927 21:30:10.054737 14886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:30:10.063171 14886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:30:10.064888 14886 catalog_manager.cc:1259] Loaded cluster ID: 707f39afb8714b80845de7fb01a75a3f
I20260927 21:30:10.065059 14886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:30:10.068321 14886 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 21:30:10.086558 14886 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 21:30:10.094904 14886 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:30:10.096259 14886 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751: Loaded TSK: 0
I20260927 21:30:10.097419 14886 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:30:10.356235 14879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:30:10.356580 14879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:30:10.356781 14879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:30:10.364195 14879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:30:10.364364 14879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:30:10.364548 14879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:30:10.374709 14879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45047
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42881
--tserver_master_addrs=127.0.97.62:37797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:30:10.376058 14879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:30:10.377410 14879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:30:10.392613 14905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:10.392851 14907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:10.393332 14904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:30:10.393659 14879 server_base.cc:1034] running on GCE node
I20260927 21:30:10.394511 14879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:30:10.395821 14879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:30:10.397086 14879 hybrid_clock.cc:648] HybridClock initialized: now 1790544610397033 us; error 40 us; skew 500 ppm
I20260927 21:30:10.397488 14879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:30:10 dist-test-slave-bxm0 krb5kdc[14257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:30:10.401541 14879 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 21:30:10.403256 14879 webserver.cc:460] Webserver started at http://127.0.97.1:42881/ using document root <none> and password file <none>
I20260927 21:30:10.404023 14879 fs_manager.cc:359] Metadata directory not provided
I20260927 21:30:10.404188 14879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:30:10.409904 14879 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 21:30:10.412801 14914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:10.414254 14879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:30:10.414430 14879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "f92ecd52d3014aeabccda938a17e7e68"
format_stamp: "Formatted at 2026-09-27 21:30:07 on dist-test-slave-bxm0"
server_key: "55d24bccfd8ecfcbe8cc905254bc434b"
server_key_iv: "fc1e842703c8bf9739f43cfa3bae52e7"
server_key_version: "encryptionkey@0"
I20260927 21:30:10.415443 14879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:30:10.437199 14879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:30:10.442256 14879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:30:10.442539 14879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:30:10.443581 14879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:30:10.445001 14879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:30:10.445123 14879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:10.445247 14879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:30:10.445333 14879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:10.477875 14879 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45047
I20260927 21:30:10.477941 15026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45047 every 8 connection(s)
I20260927 21:30:10.479419 14879 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:30:10.482491 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 14879
I20260927 21:30:10.483310 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:41425
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37797
--webserver_port=42471
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:30:10 dist-test-slave-bxm0 krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:30:10.506297 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:10.482845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.1:45779)
0927 21:30:10.483707 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:10.483758 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:10.485543 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:10.487345 (+ 1802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:10.487352 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:10.487883 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:30:10.488422 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:10.488439 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:10.490587 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:10.490595 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:10.492882 (+ 2287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:10.492908 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:10.494040 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:10.494049 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:10.494360 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:10.495080 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:30:10.495117 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:30:10.497980 (+ 2863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:30:10.501697 (+ 3717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:10.501706 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:10.501724 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:10.502271 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:10.502746 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:10.502751 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:10.502754 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:10.502944 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:10.503703 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:30:10.503712 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:30:10.504065 (+ 353us) client_negotiation.cc:769] Sending connection context
0927 21:30:10.504331 (+ 266us) client_negotiation.cc:240] Negotiation successful
0927 21:30:10.504902 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":365,"threads_started":1}
I20260927 21:30:10.506297 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:10.482447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45779 (local address 127.0.97.62:37797)
0927 21:30:10.483136 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:10.483153 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:10.483910 (+ 757us) server_negotiation.cc:407] Connection header received
0927 21:30:10.486253 (+ 2343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:10.486275 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:10.486789 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:10.487224 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:10.488668 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:10.490414 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:10.493042 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:10.493788 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:10.493925 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:10.498306 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:10.498315 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:10.498332 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:10.498380 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:10.501502 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:10.502432 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:10.502441 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:10.502447 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:10.502613 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:10.503074 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:10.503079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:10.503081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:10.503563 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:10.503769 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:10.504550 (+ 781us) server_negotiation.cc:299] Negotiation successful
0927 21:30:10.504931 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":291,"threads_started":1}
I20260927 21:30:10.509428 15027 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37797
I20260927 21:30:10.509776 15027 heartbeater.cc:464] Registering TS with master...
I20260927 21:30:10.510542 15027 heartbeater.cc:511] Master 127.0.97.62:37797 requested a full tablet report, sending...
I20260927 21:30:10.512733 14842 ts_manager.cc:194] Registered new tserver with Master: f92ecd52d3014aeabccda938a17e7e68 (127.0.97.1:45047)
I20260927 21:30:10.514243 14842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45779
W20260927 21:30:10.818387 15031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:30:10.818676 15031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:30:10.818878 15031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:30:10.826274 15031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:30:10.826395 15031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:30:10.826588 15031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:30:10.836802 15031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:41425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=42471
--tserver_master_addrs=127.0.97.62:37797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:30:10.838254 15031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:30:10.839670 15031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:30:10.852949 15036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:10.852990 15037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:10.853035 15039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:30:10.853922 15031 server_base.cc:1034] running on GCE node
I20260927 21:30:10.854573 15031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:30:10.855535 15031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:30:10.856842 15031 hybrid_clock.cc:648] HybridClock initialized: now 1790544610856778 us; error 40 us; skew 500 ppm
I20260927 21:30:10.857170 15031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:30:10 dist-test-slave-bxm0 krb5kdc[14257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:30:10.860375 15031 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 21:30:10.862023 15031 webserver.cc:460] Webserver started at http://127.0.97.2:42471/ using document root <none> and password file <none>
I20260927 21:30:10.862746 15031 fs_manager.cc:359] Metadata directory not provided
I20260927 21:30:10.862908 15031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:30:10.868337 15031 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 21:30:10.871137 15046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:10.872500 15031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:30:10.872689 15031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "bfe26c8bab6e40c49bcde52dca387731"
format_stamp: "Formatted at 2026-09-27 21:30:07 on dist-test-slave-bxm0"
server_key: "da7eed9b93593dc5d02b8ec08e52737c"
server_key_iv: "4f6f66f7340b738d223977c1d20231f3"
server_key_version: "encryptionkey@0"
I20260927 21:30:10.873751 15031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:30:10.898923 15031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:30:10.903678 15031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:30:10.903988 15031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:30:10.905045 15031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:30:10.906574 15031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:30:10.906672 15031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:10.906831 15031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:30:10.906903 15031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:10.941352 15031 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41425
I20260927 21:30:10.941416 15158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41425 every 8 connection(s)
I20260927 21:30:10.942962 15031 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:30:10.949057 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 15031
I20260927 21:30:10.949733 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:41909
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37797
--webserver_port=44509
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:30:10 dist-test-slave-bxm0 krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:30:10.963312 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:10.945505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44605 (local address 127.0.97.62:37797)
0927 21:30:10.945719 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:10.945724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:10.946900 (+ 1176us) server_negotiation.cc:407] Connection header received
0927 21:30:10.948795 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:10.948802 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:10.948902 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:10.949016 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:10.950704 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:10.951401 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:10.952964 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:10.953395 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:10.953526 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:10.957224 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:10.957230 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:10.957233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:10.957267 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:10.959934 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:10.960872 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:10.960884 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:10.960887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:10.961010 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:10.961465 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:10.961472 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:10.961479 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:10.961820 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:10.961992 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:10.962711 (+ 719us) server_negotiation.cc:299] Negotiation successful
0927 21:30:10.962964 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:30:10.964362 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:10.946014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.2:44605)
0927 21:30:10.946723 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:10.946766 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:10.948490 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:10.949301 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:10.949311 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:10.949936 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:30:10.950526 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:10.950543 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:10.951556 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:10.951566 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:10.952799 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:10.952810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:10.953661 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:10.953670 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:10.953903 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:10.954603 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:30:10.954626 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:30:10.956948 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:30:10.960095 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:10.960105 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:10.960123 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:10.960737 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:10.961132 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:10.961137 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:10.961140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:10.961340 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:10.961948 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:30:10.961959 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:30:10.962334 (+ 375us) client_negotiation.cc:769] Sending connection context
0927 21:30:10.962605 (+ 271us) client_negotiation.cc:240] Negotiation successful
0927 21:30:10.963176 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":276,"threads_started":1}
I20260927 21:30:10.965960 15159 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37797
I20260927 21:30:10.966277 15159 heartbeater.cc:464] Registering TS with master...
I20260927 21:30:10.966987 15159 heartbeater.cc:511] Master 127.0.97.62:37797 requested a full tablet report, sending...
I20260927 21:30:10.968571 14842 ts_manager.cc:194] Registered new tserver with Master: bfe26c8bab6e40c49bcde52dca387731 (127.0.97.2:41425)
I20260927 21:30:10.969944 14842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44605
W20260927 21:30:11.282895 15162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:30:11.283186 15162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:30:11.283281 15162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:30:11.290627 15162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:30:11.290752 15162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:30:11.290866 15162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:30:11.300827 15162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:41909
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=44509
--tserver_master_addrs=127.0.97.62:37797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:30:11.302150 15162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:30:11.303396 15162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:30:11.317189 15167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:11.317200 15168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:11.317234 15170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:30:11.318120 15162 server_base.cc:1034] running on GCE node
I20260927 21:30:11.318789 15162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:30:11.319787 15162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:30:11.320992 15162 hybrid_clock.cc:648] HybridClock initialized: now 1790544611320950 us; error 35 us; skew 500 ppm
I20260927 21:30:11.321262 15162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:30:11 dist-test-slave-bxm0 krb5kdc[14257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:30:11.324579 15162 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 21:30:11.326231 15162 webserver.cc:460] Webserver started at http://127.0.97.3:44509/ using document root <none> and password file <none>
I20260927 21:30:11.326946 15162 fs_manager.cc:359] Metadata directory not provided
I20260927 21:30:11.327107 15162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:30:11.332788 15162 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:30:11.335637 15177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:11.337016 15162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:30:11.337199 15162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "b0ae63d02c9345f996a817c27132fb72"
format_stamp: "Formatted at 2026-09-27 21:30:08 on dist-test-slave-bxm0"
server_key: "8842a7693551bcd9dabad6498a38885a"
server_key_iv: "b2e8fa57a55481976f477e6953432d67"
server_key_version: "encryptionkey@0"
I20260927 21:30:11.338256 15162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:30:11.360441 15162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:30:11.364737 15162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:30:11.365044 15162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:30:11.366057 15162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:30:11.367461 15162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:30:11.367581 15162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:11.367731 15162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:30:11.367820 15162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:11.400921 15162 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41909
I20260927 21:30:11.401010 15289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41909 every 8 connection(s)
I20260927 21:30:11.402460 15162 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:30:11.404075 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 15162
Sep 27 21:30:11 dist-test-slave-bxm0 krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:30:11.421237 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:11.405152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52068 (local address 127.0.97.62:37797)
0927 21:30:11.405411 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:11.405421 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:11.405993 (+ 572us) server_negotiation.cc:407] Connection header received
0927 21:30:11.406279 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:11.406283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:11.406345 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:11.406468 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:11.407969 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.408728 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.410733 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.411410 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.411531 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:11.414784 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:11.414791 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:11.414797 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:11.414836 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:11.418128 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:11.419121 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:11.419127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:11.419129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:11.419257 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:11.419641 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:11.419648 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:11.419651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:11.420055 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:11.420327 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:11.420632 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 21:30:11.420906 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":77}
I20260927 21:30:11.425468 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:11.405198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33311 (local address 127.0.97.62:37797)
0927 21:30:11.406106 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:11.406111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:11.406957 (+ 846us) server_negotiation.cc:407] Connection header received
0927 21:30:11.408768 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:11.408776 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:11.408858 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:11.408946 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:11.410953 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.412020 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.414507 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.415169 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.415287 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:11.418948 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:11.418952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:11.418955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:11.418984 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:11.422477 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:11.423220 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:11.423226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:11.423229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:11.423342 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:11.423770 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:11.423774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:11.423778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:11.424038 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:11.424188 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:11.424878 (+ 690us) server_negotiation.cc:299] Negotiation successful
0927 21:30:11.425080 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":806,"thread_start_us":680,"threads_started":1}
I20260927 21:30:11.426442 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:11.405954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.3:33311)
0927 21:30:11.406789 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:11.406833 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:11.408526 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:11.409255 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:11.409270 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:11.410022 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:30:11.410798 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:11.410818 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.412176 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.412202 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:11.414357 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:11.414373 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.415377 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.415382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:11.415580 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:11.416108 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:30:11.416134 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:30:11.418646 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:30:11.422625 (+ 3979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:11.422641 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:11.422656 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:11.423091 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:11.423460 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:11.423465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:11.423468 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:11.423650 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:11.424163 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:30:11.424172 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:30:11.424534 (+ 362us) client_negotiation.cc:769] Sending connection context
0927 21:30:11.424792 (+ 258us) client_negotiation.cc:240] Negotiation successful
0927 21:30:11.425330 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":356,"threads_started":1}
I20260927 21:30:11.427716 15290 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37797
I20260927 21:30:11.428026 15290 heartbeater.cc:464] Registering TS with master...
I20260927 21:30:11.428725 15290 heartbeater.cc:511] Master 127.0.97.62:37797 requested a full tablet report, sending...
I20260927 21:30:11.430099 14842 ts_manager.cc:194] Registered new tserver with Master: b0ae63d02c9345f996a817c27132fb72 (127.0.97.3:41909)
I20260927 21:30:11.430900 14842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33311
I20260927 21:30:11.435950 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:30:11.443136 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:11.436837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52072 (local address 127.0.97.62:37797)
0927 21:30:11.437025 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:11.437034 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:11.437497 (+ 463us) server_negotiation.cc:407] Connection header received
0927 21:30:11.437726 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:11.437730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:11.437856 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:11.437981 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:30:11.438451 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.439131 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.440688 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.441166 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.441284 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:11.441885 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:30:11.442497 (+ 612us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:30:11.442611 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:11.442746 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 21:30:11.442836 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:30:11.444077 14842 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:52072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:30:11.451450 14842 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:30:11.475730 15300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:11.463071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45047 (local address 127.0.0.1:41458)
0927 21:30:11.465232 (+ 2161us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:11.465269 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:11.465472 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:11.467446 (+ 1974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:11.467458 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:11.467500 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:30:11.468741 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:11.468754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.470937 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.470944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:11.472739 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:11.472754 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.474657 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.474664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:11.474876 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:11.474927 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 21:30:11.475069 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 21:30:11.475275 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1877,"mutex_wait_us":783,"thread_start_us":1719,"threads_started":1}
I20260927 21:30:11.475943 15302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:11.463346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41425 (local address 127.0.0.1:38234)
0927 21:30:11.466459 (+ 3113us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:11.466479 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:11.466619 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:11.467725 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:11.467732 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:11.467767 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:30:11.468332 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:11.468348 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.470552 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.470557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:11.473167 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:11.473180 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.475024 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.475036 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:11.475210 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:11.475249 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 21:30:11.475372 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 21:30:11.475502 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2984,"mutex_wait_us":30,"thread_start_us":2849,"threads_started":1}
I20260927 21:30:11.476243 15303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:11.464010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38234 (local address 127.0.97.2:41425)
0927 21:30:11.466032 (+ 2022us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:11.466040 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:11.466766 (+ 726us) server_negotiation.cc:407] Connection header received
0927 21:30:11.466888 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:11.466913 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:11.467201 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:11.467411 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:30:11.468584 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.470384 (+ 1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.473346 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.474517 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.474645 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:11.474857 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:11.475424 (+ 567us) server_negotiation.cc:299] Negotiation successful
0927 21:30:11.475593 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1857,"thread_start_us":1752,"threads_started":1}
I20260927 21:30:11.476991 15301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:11.463331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41458 (local address 127.0.97.1:45047)
0927 21:30:11.464710 (+ 1379us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:11.464723 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:11.465473 (+ 750us) server_negotiation.cc:407] Connection header received
0927 21:30:11.465740 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:11.465752 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:11.466058 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:11.467914 (+ 1856us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:30:11.469238 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.470761 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.472904 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.474425 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.476234 (+ 1809us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:11.476442 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:11.476511 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 21:30:11.476681 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1180,"thread_start_us":1042,"threads_started":1}
I20260927 21:30:11.479001 15304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:11.466354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41909 (local address 127.0.0.1:36076)
0927 21:30:11.469232 (+ 2878us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:11.469249 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:11.469353 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:11.470518 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:11.470523 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:11.470546 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:30:11.471123 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:11.471135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.473158 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.473163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:11.475184 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:11.475196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.476967 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.476973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:11.478443 (+ 1470us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:11.478481 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 21:30:11.478571 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 21:30:11.478698 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2777,"thread_start_us":2657,"threads_started":1}
I20260927 21:30:11.480036 15305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:11.466816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36076 (local address 127.0.97.3:41909)
0927 21:30:11.469941 (+ 3125us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:11.469949 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:11.469979 (+ 30us) server_negotiation.cc:407] Connection header received
0927 21:30:11.470063 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:11.470069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:11.470297 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:11.471095 (+ 798us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:30:11.471399 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.473013 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.475349 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.476753 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.476881 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:11.477122 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:11.479523 (+ 2401us) server_negotiation.cc:299] Negotiation successful
0927 21:30:11.479693 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2954,"thread_start_us":2780,"threads_started":1}
I20260927 21:30:11.480450 15094 tablet_service.cc:1467] Processing CreateTablet for tablet 86864a1aa8894b2f9524d81c89e637d3 (DEFAULT_TABLE table=test-table [id=f8f5743efecd489fa078fe397223bda6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:30:11.480948 14962 tablet_service.cc:1467] Processing CreateTablet for tablet 86864a1aa8894b2f9524d81c89e637d3 (DEFAULT_TABLE table=test-table [id=f8f5743efecd489fa078fe397223bda6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:30:11.482050 15094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86864a1aa8894b2f9524d81c89e637d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:30:11.483651 14962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86864a1aa8894b2f9524d81c89e637d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:30:11.483839 15225 tablet_service.cc:1467] Processing CreateTablet for tablet 86864a1aa8894b2f9524d81c89e637d3 (DEFAULT_TABLE table=test-table [id=f8f5743efecd489fa078fe397223bda6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:30:11.485162 15225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86864a1aa8894b2f9524d81c89e637d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:30:11.497310 15306 tablet_bootstrap.cc:492] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68: Bootstrap starting.
I20260927 21:30:11.499364 15307 tablet_bootstrap.cc:492] T 86864a1aa8894b2f9524d81c89e637d3 P bfe26c8bab6e40c49bcde52dca387731: Bootstrap starting.
I20260927 21:30:11.500821 15308 tablet_bootstrap.cc:492] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72: Bootstrap starting.
I20260927 21:30:11.501113 15306 tablet_bootstrap.cc:654] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68: Neither blocks nor log segments found. Creating new log.
I20260927 21:30:11.502552 15306 log.cc:824] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68: Log is configured to *not* fsync() on all Append() calls
I20260927 21:30:11.503149 15307 tablet_bootstrap.cc:654] T 86864a1aa8894b2f9524d81c89e637d3 P bfe26c8bab6e40c49bcde52dca387731: Neither blocks nor log segments found. Creating new log.
I20260927 21:30:11.503821 15308 tablet_bootstrap.cc:654] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72: Neither blocks nor log segments found. Creating new log.
I20260927 21:30:11.504537 15307 log.cc:824] T 86864a1aa8894b2f9524d81c89e637d3 P bfe26c8bab6e40c49bcde52dca387731: Log is configured to *not* fsync() on all Append() calls
I20260927 21:30:11.505010 15308 log.cc:824] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72: Log is configured to *not* fsync() on all Append() calls
I20260927 21:30:11.506013 15306 tablet_bootstrap.cc:492] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68: No bootstrap required, opened a new log
I20260927 21:30:11.506356 15306 ts_tablet_manager.cc:1397] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260927 21:30:11.508482 15307 tablet_bootstrap.cc:492] T 86864a1aa8894b2f9524d81c89e637d3 P bfe26c8bab6e40c49bcde52dca387731: No bootstrap required, opened a new log
I20260927 21:30:11.508482 15308 tablet_bootstrap.cc:492] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72: No bootstrap required, opened a new log
I20260927 21:30:11.508751 15308 ts_tablet_manager.cc:1397] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260927 21:30:11.508751 15307 ts_tablet_manager.cc:1397] T 86864a1aa8894b2f9524d81c89e637d3 P bfe26c8bab6e40c49bcde52dca387731: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20260927 21:30:11.514012 15306 raft_consensus.cc:348] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 } } peers { permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } } peers { permanent_uuid: "b0ae63d02c9345f996a817c27132fb72" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41909 } }
I20260927 21:30:11.514385 15306 raft_consensus.cc:374] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:30:11.514515 15306 raft_consensus.cc:729] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f92ecd52d3014aeabccda938a17e7e68, State: Initialized, Role: FOLLOWER
I20260927 21:30:11.515152 15306 consensus_queue.cc:260] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 } } peers { permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } } peers { permanent_uuid: "b0ae63d02c9345f996a817c27132fb72" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41909 } }
I20260927 21:30:11.517176 15027 heartbeater.cc:503] Master 127.0.97.62:37797 was elected leader, sending a full tablet report...
I20260927 21:30:11.518364 15306 ts_tablet_manager.cc:1428] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260927 21:30:11.519610 15307 raft_consensus.cc:348] T 86864a1aa8894b2f9524d81c89e637d3 P bfe26c8bab6e40c49bcde52dca387731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 } } peers { permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } } peers { permanent_uuid: "b0ae63d02c9345f996a817c27132fb72" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41909 } }
I20260927 21:30:11.520069 15307 raft_consensus.cc:374] T 86864a1aa8894b2f9524d81c89e637d3 P bfe26c8bab6e40c49bcde52dca387731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:30:11.519868 15308 raft_consensus.cc:348] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 } } peers { permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } } peers { permanent_uuid: "b0ae63d02c9345f996a817c27132fb72" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41909 } }
I20260927 21:30:11.520174 15307 raft_consensus.cc:729] T 86864a1aa8894b2f9524d81c89e637d3 P bfe26c8bab6e40c49bcde52dca387731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfe26c8bab6e40c49bcde52dca387731, State: Initialized, Role: FOLLOWER
I20260927 21:30:11.520282 15308 raft_consensus.cc:374] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:30:11.520372 15308 raft_consensus.cc:729] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ae63d02c9345f996a817c27132fb72, State: Initialized, Role: FOLLOWER
I20260927 21:30:11.520810 15307 consensus_queue.cc:260] T 86864a1aa8894b2f9524d81c89e637d3 P bfe26c8bab6e40c49bcde52dca387731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 } } peers { permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } } peers { permanent_uuid: "b0ae63d02c9345f996a817c27132fb72" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41909 } }
I20260927 21:30:11.520982 15308 consensus_queue.cc:260] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 } } peers { permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } } peers { permanent_uuid: "b0ae63d02c9345f996a817c27132fb72" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41909 } }
I20260927 21:30:11.521971 15159 heartbeater.cc:503] Master 127.0.97.62:37797 was elected leader, sending a full tablet report...
I20260927 21:30:11.522843 15290 heartbeater.cc:503] Master 127.0.97.62:37797 was elected leader, sending a full tablet report...
I20260927 21:30:11.523957 15307 ts_tablet_manager.cc:1428] T 86864a1aa8894b2f9524d81c89e637d3 P bfe26c8bab6e40c49bcde52dca387731: Time spent starting tablet: real 0.015s user 0.006s sys 0.009s
I20260927 21:30:11.524224 15308 ts_tablet_manager.cc:1428] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260927 21:30:11.648801 15314 raft_consensus.cc:482] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:30:11.649113 15314 raft_consensus.cc:504] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 } } peers { permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } } peers { permanent_uuid: "b0ae63d02c9345f996a817c27132fb72" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41909 } }
I20260927 21:30:11.650861 15314 leader_election.cc:288] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f92ecd52d3014aeabccda938a17e7e68 (127.0.97.1:45047), bfe26c8bab6e40c49bcde52dca387731 (127.0.97.2:41425)
W20260927 21:30:11.654021 15291 tablet.cc:2369] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:30:11.657487 15301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:11.651327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50423 (local address 127.0.97.1:45047)
0927 21:30:11.651515 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:11.651521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:11.651543 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:30:11.651625 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:11.651633 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:11.651722 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:11.651826 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:30:11.652308 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.653288 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.655312 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.656553 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.656684 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:11.656793 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:11.657095 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 21:30:11.657232 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:30:11.657691 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:11.651144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45047 (local address 127.0.97.3:50423)
0927 21:30:11.651345 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:11.651366 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:11.651496 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:11.651874 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:11.651883 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:11.651919 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:30:11.652104 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:11.652112 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.653420 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.653426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:11.655154 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:11.655168 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.656743 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.656748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:11.656923 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:11.656954 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 21:30:11.657052 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 21:30:11.657200 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":52}
I20260927 21:30:11.658802 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:11.651149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41425 (local address 127.0.97.3:39565)
0927 21:30:11.652106 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:11.652120 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:11.652243 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:11.652545 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:11.652550 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:11.652576 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:30:11.652906 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:11.652917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.654302 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.654311 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:11.656513 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:11.656528 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.658157 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.658163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:11.658340 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:11.658375 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 21:30:11.658463 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 21:30:11.658552 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":236,"thread_start_us":329,"threads_started":1}
I20260927 21:30:11.658911 15303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:11.651346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39565 (local address 127.0.97.2:41425)
0927 21:30:11.651582 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:11.651591 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:11.652215 (+ 624us) server_negotiation.cc:407] Connection header received
0927 21:30:11.652366 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:11.652370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:11.652430 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:11.652498 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:30:11.653051 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.654020 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.656707 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.657952 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.658081 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:11.658158 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:11.658518 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 21:30:11.658654 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 21:30:11.658905 14982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86864a1aa8894b2f9524d81c89e637d3" candidate_uuid: "b0ae63d02c9345f996a817c27132fb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f92ecd52d3014aeabccda938a17e7e68" is_pre_election: true
I20260927 21:30:11.659508 14982 raft_consensus.cc:2454] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0ae63d02c9345f996a817c27132fb72 in term 0.
I20260927 21:30:11.659991 15114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86864a1aa8894b2f9524d81c89e637d3" candidate_uuid: "b0ae63d02c9345f996a817c27132fb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe26c8bab6e40c49bcde52dca387731" is_pre_election: true
I20260927 21:30:11.660302 15180 leader_election.cc:302] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0ae63d02c9345f996a817c27132fb72, f92ecd52d3014aeabccda938a17e7e68; no voters:
I20260927 21:30:11.660523 15114 raft_consensus.cc:2454] T 86864a1aa8894b2f9524d81c89e637d3 P bfe26c8bab6e40c49bcde52dca387731 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0ae63d02c9345f996a817c27132fb72 in term 0.
I20260927 21:30:11.660745 15314 raft_consensus.cc:2781] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:30:11.660907 15314 raft_consensus.cc:482] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:30:11.661019 15314 raft_consensus.cc:3037] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:30:11.663278 15314 raft_consensus.cc:504] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 } } peers { permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } } peers { permanent_uuid: "b0ae63d02c9345f996a817c27132fb72" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41909 } }
I20260927 21:30:11.663957 15314 leader_election.cc:288] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72 [CANDIDATE]: Term 1 election: Requested vote from peers f92ecd52d3014aeabccda938a17e7e68 (127.0.97.1:45047), bfe26c8bab6e40c49bcde52dca387731 (127.0.97.2:41425)
I20260927 21:30:11.664458 14982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86864a1aa8894b2f9524d81c89e637d3" candidate_uuid: "b0ae63d02c9345f996a817c27132fb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f92ecd52d3014aeabccda938a17e7e68"
I20260927 21:30:11.664523 15114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86864a1aa8894b2f9524d81c89e637d3" candidate_uuid: "b0ae63d02c9345f996a817c27132fb72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe26c8bab6e40c49bcde52dca387731"
I20260927 21:30:11.664660 14982 raft_consensus.cc:3037] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:30:11.664781 15114 raft_consensus.cc:3037] T 86864a1aa8894b2f9524d81c89e637d3 P bfe26c8bab6e40c49bcde52dca387731 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:30:11.667090 14982 raft_consensus.cc:2454] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0ae63d02c9345f996a817c27132fb72 in term 1.
I20260927 21:30:11.667105 15114 raft_consensus.cc:2454] T 86864a1aa8894b2f9524d81c89e637d3 P bfe26c8bab6e40c49bcde52dca387731 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0ae63d02c9345f996a817c27132fb72 in term 1.
I20260927 21:30:11.667507 15180 leader_election.cc:302] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0ae63d02c9345f996a817c27132fb72, f92ecd52d3014aeabccda938a17e7e68; no voters:
I20260927 21:30:11.667835 15314 raft_consensus.cc:2781] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:30:11.668310 15314 raft_consensus.cc:686] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72 [term 1 LEADER]: Becoming Leader. State: Replica: b0ae63d02c9345f996a817c27132fb72, State: Running, Role: LEADER
I20260927 21:30:11.668851 15314 consensus_queue.cc:237] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 } } peers { permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } } peers { permanent_uuid: "b0ae63d02c9345f996a817c27132fb72" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41909 } }
I20260927 21:30:11.673595 14842 catalog_manager.cc:5495] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0ae63d02c9345f996a817c27132fb72 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "b0ae63d02c9345f996a817c27132fb72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0ae63d02c9345f996a817c27132fb72" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41909 } health_report { overall_health: HEALTHY } } }
W20260927 21:30:11.695052 15160 tablet.cc:2369] T 86864a1aa8894b2f9524d81c89e637d3 P bfe26c8bab6e40c49bcde52dca387731: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:30:11.715119 15305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:11.709773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36088 (local address 127.0.97.3:41909)
0927 21:30:11.709961 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:11.709965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:11.710083 (+ 118us) server_negotiation.cc:407] Connection header received
0927 21:30:11.710284 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:11.710287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:11.710373 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:11.710509 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:30:11.710857 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.711617 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.712974 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.713349 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.713444 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:11.714059 (+ 615us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:30:11.714658 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:30:11.714786 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:11.714851 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 21:30:11.714918 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:30:11.723078 15114 raft_consensus.cc:1261] T 86864a1aa8894b2f9524d81c89e637d3 P bfe26c8bab6e40c49bcde52dca387731 [term 1 FOLLOWER]: Refusing update from remote peer b0ae63d02c9345f996a817c27132fb72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:30:11.723115 14982 raft_consensus.cc:1261] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [term 1 FOLLOWER]: Refusing update from remote peer b0ae63d02c9345f996a817c27132fb72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:30:11.724036 15316 consensus_queue.cc:1035] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:30:11.724372 15314 consensus_queue.cc:1035] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 21:30:11.732079 15028 tablet.cc:2369] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:30:11.744499 15319 mvcc.cc:205] Tried to move back new op lower bound from 7334070729608458240 to 7334070729399697408. Current Snapshot: MvccSnapshot[applied={T|T < 7334070729608458240}]
I20260927 21:30:11.750797 14841 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:52072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:30:11.751148 14841 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:52072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:30:11.755713 14841 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751: Sending DeleteTablet for 3 replicas of tablet 86864a1aa8894b2f9524d81c89e637d3
I20260927 21:30:11.757112 15225 tablet_service.cc:1514] Processing DeleteTablet for tablet 86864a1aa8894b2f9524d81c89e637d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:30:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36076
I20260927 21:30:11.757432 14962 tablet_service.cc:1514] Processing DeleteTablet for tablet 86864a1aa8894b2f9524d81c89e637d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:30:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41458
I20260927 21:30:11.757473 15094 tablet_service.cc:1514] Processing DeleteTablet for tablet 86864a1aa8894b2f9524d81c89e637d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:30:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38234
I20260927 21:30:11.758471 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 14879
I20260927 21:30:11.767030 15330 tablet_replica.cc:330] T 86864a1aa8894b2f9524d81c89e637d3 P bfe26c8bab6e40c49bcde52dca387731: stopping tablet replica
I20260927 21:30:11.768224 15330 raft_consensus.cc:2229] T 86864a1aa8894b2f9524d81c89e637d3 P bfe26c8bab6e40c49bcde52dca387731 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:30:11.769555 15330 raft_consensus.cc:2258] T 86864a1aa8894b2f9524d81c89e637d3 P bfe26c8bab6e40c49bcde52dca387731 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:30:11.773253 15330 ts_tablet_manager.cc:1905] T 86864a1aa8894b2f9524d81c89e637d3 P bfe26c8bab6e40c49bcde52dca387731: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:30:11.779570 15329 tablet_replica.cc:330] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68: stopping tablet replica
I20260927 21:30:11.780717 15329 raft_consensus.cc:2229] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:30:11.781945 15329 raft_consensus.cc:2258] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:30:11.783720 15301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:11.760401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41466 (local address 127.0.97.1:45047)
0927 21:30:11.760618 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:11.760626 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:11.760649 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:30:11.760718 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:11.760726 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:11.760833 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:11.760951 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:11.763588 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.765156 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.767513 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.768233 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.768387 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:11.773124 (+ 4737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:11.773144 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:11.773161 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:11.773209 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:11.778061 (+ 4852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:11.779435 (+ 1374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:11.779447 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:11.779450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:11.779616 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:11.780214 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:11.780222 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:11.780228 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:11.780801 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:11.781171 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:11.782678 (+ 1507us) server_negotiation.cc:299] Negotiation successful
0927 21:30:11.783206 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 21:30:11.784456 15330 ts_tablet_manager.cc:1918] T 86864a1aa8894b2f9524d81c89e637d3 P bfe26c8bab6e40c49bcde52dca387731: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:30:11.784669 15330 log.cc:1196] T 86864a1aa8894b2f9524d81c89e637d3 P bfe26c8bab6e40c49bcde52dca387731: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/wal/wals/86864a1aa8894b2f9524d81c89e637d3
I20260927 21:30:11.784914 15022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:30:11.785259 15330 ts_tablet_manager.cc:1939] T 86864a1aa8894b2f9524d81c89e637d3 P bfe26c8bab6e40c49bcde52dca387731: Deleting consensus metadata
I20260927 21:30:11.785881 15328 tablet_replica.cc:330] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72: stopping tablet replica
I20260927 21:30:11.786546 15328 raft_consensus.cc:2229] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:30:11.787381 14827 catalog_manager.cc:4841] TS bfe26c8bab6e40c49bcde52dca387731 (127.0.97.2:41425): tablet 86864a1aa8894b2f9524d81c89e637d3 (table test-table [id=f8f5743efecd489fa078fe397223bda6]) successfully deleted
I20260927 21:30:11.788173 15328 raft_consensus.cc:2258] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:30:11.790275 15328 ts_tablet_manager.cc:1905] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:30:11.797865 15328 ts_tablet_manager.cc:1918] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:30:11.798065 15328 log.cc:1196] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/wal/wals/86864a1aa8894b2f9524d81c89e637d3
I20260927 21:30:11.798677 15328 ts_tablet_manager.cc:1939] T 86864a1aa8894b2f9524d81c89e637d3 P b0ae63d02c9345f996a817c27132fb72: Deleting consensus metadata
I20260927 21:30:11.800572 14828 catalog_manager.cc:4841] TS b0ae63d02c9345f996a817c27132fb72 (127.0.97.3:41909): tablet 86864a1aa8894b2f9524d81c89e637d3 (table test-table [id=f8f5743efecd489fa078fe397223bda6]) successfully deleted
I20260927 21:30:11.879878 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 14879
I20260927 21:30:11.904085 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 15031
W20260927 21:30:11.904203 14828 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45047 (error 108)
I20260927 21:30:11.907791 15304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:11.907363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45047 (local address 127.0.0.1:41478)
0927 21:30:11.907542 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:11.907651 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260927 21:30:11.908288 14828 catalog_manager.cc:4584] TS f92ecd52d3014aeabccda938a17e7e68 (127.0.97.1:45047): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 86864a1aa8894b2f9524d81c89e637d3: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45047: connect: Connection refused (error 111)
I20260927 21:30:11.920019 15303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:11.905291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38236 (local address 127.0.97.2:41425)
0927 21:30:11.905494 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:11.905500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:11.905521 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:30:11.905584 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:11.905590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:11.905701 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:11.905808 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:11.907245 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.908418 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.910493 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:11.910886 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:11.910989 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:11.914015 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:11.914029 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:11.914044 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:11.914092 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:11.916959 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:11.917781 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:11.917798 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:11.917801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:11.917976 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:11.918442 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:11.918445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:11.918447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:11.918921 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:11.919144 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:11.919543 (+ 399us) server_negotiation.cc:299] Negotiation successful
0927 21:30:11.919711 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 21:30:11.920694 15154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:30:11.962086 15304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:11.961585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45047 (local address 127.0.0.1:41484)
0927 21:30:11.961814 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:11.961936 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 21:30:11.963148 15304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:11.962801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45047 (local address 127.0.0.1:41496)
0927 21:30:11.962950 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:11.963030 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 21:30:12.023813 15304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:12.023224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45047 (local address 127.0.0.1:41508)
0927 21:30:12.023496 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:12.023610 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 21:30:12.024992 15304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:12.024633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45047 (local address 127.0.0.1:41520)
0927 21:30:12.024767 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:12.024851 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 21:30:12.033186 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 15031
I20260927 21:30:12.051964 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 15162
I20260927 21:30:12.067674 15305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:12.052862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36094 (local address 127.0.97.3:41909)
0927 21:30:12.053060 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:12.053066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:12.053086 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:30:12.053187 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:12.053193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:12.053268 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:12.053392 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:12.054719 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:12.055574 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:12.056958 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:12.057669 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:12.057813 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:12.061315 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:12.061325 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:12.061340 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:12.061383 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:12.064545 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:12.065472 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:12.065478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:12.065480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:12.065728 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:12.066170 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:12.066173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:12.066175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:12.066518 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:12.066753 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:12.067127 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 21:30:12.067296 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 21:30:12.068478 15285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:30:12.105950 15304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:12.105363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45047 (local address 127.0.0.1:41534)
0927 21:30:12.105634 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:12.105763 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260927 21:30:12.107227 15304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:12.106748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45047 (local address 127.0.0.1:41544)
0927 21:30:12.107025 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:12.107099 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260927 21:30:12.189579 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 15162
I20260927 21:30:12.208666 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 14809
I20260927 21:30:12.209882 14872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:30:12.322980 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 14809
I20260927 21:30:12.345146 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37797
--webserver_interface=127.0.97.62
--webserver_port=40891
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:30:12.676261 15336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:30:12.676618 15336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:30:12.676774 15336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:30:12.684199 15336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:30:12.684360 15336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:30:12.684463 15336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:30:12.684566 15336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:30:12.684641 15336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:30:12.695096 15336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=40891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:30:12.696578 15336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:30:12.697914 15336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:30:12.709705 15341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:12.709762 15344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:12.709976 15342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:30:12.710789 15336 server_base.cc:1034] running on GCE node
I20260927 21:30:12.711647 15336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:30:12.713043 15336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:30:12.714236 15336 hybrid_clock.cc:648] HybridClock initialized: now 1790544612714197 us; error 31 us; skew 500 ppm
I20260927 21:30:12.714543 15336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:30:12 dist-test-slave-bxm0 krb5kdc[14257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:30:12.717892 15336 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:30:12.719501 15336 webserver.cc:460] Webserver started at http://127.0.97.62:40891/ using document root <none> and password file <none>
I20260927 21:30:12.720170 15336 fs_manager.cc:359] Metadata directory not provided
I20260927 21:30:12.720273 15336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:30:12.726032 15336 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 21:30:12.728924 15351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:12.730402 15336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:30:12.730595 15336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "76f94b335cdc416d9b1742c8e36dc751"
format_stamp: "Formatted at 2026-09-27 21:30:06 on dist-test-slave-bxm0"
server_key: "7f05210ad6401a58ef0ee35f900a0a14"
server_key_iv: "677e47646ef496327190ebca3c0fb114"
server_key_version: "encryptionkey@0"
I20260927 21:30:12.731608 15336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:30:12.748667 15336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:30:12.754141 15336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:30:12.754487 15336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:30:12.775113 15336 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37797
I20260927 21:30:12.775136 15402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37797 every 8 connection(s)
I20260927 21:30:12.776695 15336 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:30:12.777545 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 15336
I20260927 21:30:12.778503 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45047
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37797
--webserver_port=42881
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:30:12.783908 15403 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:30:12.787446 15403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751: Bootstrap starting.
I20260927 21:30:12.809515 15403 log.cc:824] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751: Log is configured to *not* fsync() on all Append() calls
I20260927 21:30:12.829501 15403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:30:12.830258 15403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751: Bootstrap complete.
I20260927 21:30:12.841790 15403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f94b335cdc416d9b1742c8e36dc751" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37797 } }
I20260927 21:30:12.842877 15403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f94b335cdc416d9b1742c8e36dc751, State: Initialized, Role: FOLLOWER
I20260927 21:30:12.843637 15403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f94b335cdc416d9b1742c8e36dc751" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37797 } }
I20260927 21:30:12.843959 15403 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:30:12.844130 15403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:30:12.844352 15403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [term 2 FOLLOWER]: Advancing to term 3
I20260927 21:30:12.848860 15403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f94b335cdc416d9b1742c8e36dc751" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37797 } }
I20260927 21:30:12.849455 15403 leader_election.cc:302] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76f94b335cdc416d9b1742c8e36dc751; no voters:
I20260927 21:30:12.850294 15403 leader_election.cc:288] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 21:30:12.850452 15407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 21:30:12.851815 15407 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [term 3 LEADER]: Becoming Leader. State: Replica: 76f94b335cdc416d9b1742c8e36dc751, State: Running, Role: LEADER
I20260927 21:30:12.852566 15407 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f94b335cdc416d9b1742c8e36dc751" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37797 } }
I20260927 21:30:12.852964 15403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:30:12.855616 15409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76f94b335cdc416d9b1742c8e36dc751. Latest consensus state: current_term: 3 leader_uuid: "76f94b335cdc416d9b1742c8e36dc751" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f94b335cdc416d9b1742c8e36dc751" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37797 } } }
I20260927 21:30:12.856132 15409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [sys.catalog]: This master's current role is: LEADER
I20260927 21:30:12.856854 15408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "76f94b335cdc416d9b1742c8e36dc751" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f94b335cdc416d9b1742c8e36dc751" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37797 } } }
I20260927 21:30:12.857228 15408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f94b335cdc416d9b1742c8e36dc751 [sys.catalog]: This master's current role is: LEADER
I20260927 21:30:12.860044 15413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:30:12.872916 15413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:30:12.875376 15413 catalog_manager.cc:1259] Loaded cluster ID: 707f39afb8714b80845de7fb01a75a3f
I20260927 21:30:12.875582 15413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 21:30:12.878497 15413 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7588f189ad google::LogMessage::Fail() at ??:0
@ 0x7f7588f1b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f7588f184de google::LogMessage::Flush() at ??:0
@ 0x7f7588f19a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f759e94b512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f759e948976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f759e9d0610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f759e9d0439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f758bb64345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f758bb6791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f758bb67799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f758bb42eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f758ee1c6db start_thread at ??:0
@ 0x7f7587db261f clone at ??:0
W20260927 21:30:13.175601 15405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:30:13.175891 15405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:30:13.175985 15405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:30:13.183547 15405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:30:13.183667 15405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:30:13.183892 15405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:30:13.193933 15405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45047
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42881
--tserver_master_addrs=127.0.97.62:37797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:30:13.195315 15405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:30:13.196652 15405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:30:13.210018 15448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:13.210074 15446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:13.210098 15445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:30:13.210940 15405 server_base.cc:1034] running on GCE node
I20260927 21:30:13.211539 15405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:30:13.212522 15405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:30:13.213709 15405 hybrid_clock.cc:648] HybridClock initialized: now 1790544613213648 us; error 58 us; skew 500 ppm
I20260927 21:30:13.214031 15405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:30:13 dist-test-slave-bxm0 krb5kdc[14257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:30:13.217170 15405 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 21:30:13.218806 15405 webserver.cc:460] Webserver started at http://127.0.97.1:42881/ using document root <none> and password file <none>
I20260927 21:30:13.219502 15405 fs_manager.cc:359] Metadata directory not provided
I20260927 21:30:13.219653 15405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:30:13.225066 15405 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 21:30:13.228094 15455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:13.229501 15405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 21:30:13.229722 15405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "f92ecd52d3014aeabccda938a17e7e68"
format_stamp: "Formatted at 2026-09-27 21:30:07 on dist-test-slave-bxm0"
server_key: "55d24bccfd8ecfcbe8cc905254bc434b"
server_key_iv: "fc1e842703c8bf9739f43cfa3bae52e7"
server_key_version: "encryptionkey@0"
I20260927 21:30:13.230728 15405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:30:13.248711 15405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:30:13.253154 15405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:30:13.253443 15405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:30:13.254554 15405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:30:13.256465 15462 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260927 21:30:13.262017 15405 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 21:30:13.262200 15405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260927 21:30:13.262342 15405 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 21:30:13.265472 15405 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 21:30:13.265584 15405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260927 21:30:13.265962 15462 tablet_bootstrap.cc:492] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68: Bootstrap starting.
I20260927 21:30:13.274690 15462 log.cc:824] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68: Log is configured to *not* fsync() on all Append() calls
I20260927 21:30:13.290894 15462 tablet_bootstrap.cc:492] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:30:13.291579 15462 tablet_bootstrap.cc:492] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68: Bootstrap complete.
I20260927 21:30:13.292085 15462 ts_tablet_manager.cc:1397] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68: Time spent bootstrapping tablet: real 0.026s user 0.013s sys 0.010s
I20260927 21:30:13.302255 15462 raft_consensus.cc:348] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 } } peers { permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } } peers { permanent_uuid: "b0ae63d02c9345f996a817c27132fb72" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41909 } }
I20260927 21:30:13.303911 15462 raft_consensus.cc:729] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f92ecd52d3014aeabccda938a17e7e68, State: Initialized, Role: FOLLOWER
I20260927 21:30:13.304606 15462 consensus_queue.cc:260] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 } } peers { permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } } peers { permanent_uuid: "b0ae63d02c9345f996a817c27132fb72" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41909 } }
I20260927 21:30:13.306633 15462 ts_tablet_manager.cc:1428] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260927 21:30:13.309999 15405 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45047
I20260927 21:30:13.310073 15570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45047 every 8 connection(s)
I20260927 21:30:13.311246 15405 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:30:13.315724 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 15405
I20260927 21:30:13.315840 15573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:13.314042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.1:44067)
0927 21:30:13.314813 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:13.314997 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":329,"threads_started":1}
I20260927 21:30:13.316541 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:41425
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37797
--webserver_port=42471
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:30:13.316586 15571 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37797 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37797: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
I20260927 21:30:13.317480 15573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:13.317126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.1:45439)
0927 21:30:13.317267 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:13.317331 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 21:30:13.318418 15573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:13.318066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.1:55619)
0927 21:30:13.318214 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:13.318295 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 21:30:13.318611 15571 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:30:13.654896 15574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:30:13.655200 15574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:30:13.655296 15574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:30:13.663067 15574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:30:13.663185 15574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:30:13.663306 15574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:30:13.673399 15574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:41425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=42471
--tserver_master_addrs=127.0.97.62:37797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:30:13.674754 15574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:30:13.675993 15574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:30:13.689051 15579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:13.689128 15582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:13.689148 15580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:30:13.689975 15574 server_base.cc:1034] running on GCE node
I20260927 21:30:13.690629 15574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:30:13.691640 15574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:30:13.692880 15574 hybrid_clock.cc:648] HybridClock initialized: now 1790544613692841 us; error 35 us; skew 500 ppm
I20260927 21:30:13.693223 15574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:30:13 dist-test-slave-bxm0 krb5kdc[14257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:30:13.696447 15574 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 21:30:13.697952 15574 webserver.cc:460] Webserver started at http://127.0.97.2:42471/ using document root <none> and password file <none>
I20260927 21:30:13.698654 15574 fs_manager.cc:359] Metadata directory not provided
I20260927 21:30:13.698807 15574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:30:13.704432 15574 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:30:13.707320 15589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:13.708736 15574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:30:13.708946 15574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "bfe26c8bab6e40c49bcde52dca387731"
format_stamp: "Formatted at 2026-09-27 21:30:07 on dist-test-slave-bxm0"
server_key: "da7eed9b93593dc5d02b8ec08e52737c"
server_key_iv: "4f6f66f7340b738d223977c1d20231f3"
server_key_version: "encryptionkey@0"
I20260927 21:30:13.710049 15574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:30:13.726817 15574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:30:13.731112 15574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:30:13.731419 15574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:30:13.732466 15574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:30:13.733865 15574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:30:13.733995 15574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:13.734131 15574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:30:13.734221 15574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:13.768451 15574 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41425
I20260927 21:30:13.768501 15701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41425 every 8 connection(s)
I20260927 21:30:13.770061 15574 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:30:13.774874 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:13.773027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.2:37371)
0927 21:30:13.773809 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:13.774012 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":281,"threads_started":1}
W20260927 21:30:13.775566 15702 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37797 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37797: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
I20260927 21:30:13.776363 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:13.776069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.2:35475)
0927 21:30:13.776202 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:13.776265 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 21:30:13.777182 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:13.776878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.2:36911)
0927 21:30:13.777011 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:13.777087 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 21:30:13.777419 15702 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:30:13.779937 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 15574
I20260927 21:30:13.780510 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:41909
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37797
--webserver_port=44509
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:30:14.111915 15705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:30:14.112336 15705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:30:14.112509 15705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:30:14.120102 15705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:30:14.120272 15705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:30:14.120464 15705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:30:14.130373 15705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:41909
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=44509
--tserver_master_addrs=127.0.97.62:37797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:30:14.131765 15705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:30:14.133111 15705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:30:14.146486 15710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:14.146569 15711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:14.146524 15713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:30:14.152959 15705 server_base.cc:1034] running on GCE node
I20260927 21:30:14.153600 15705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:30:14.154646 15705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:30:14.155864 15705 hybrid_clock.cc:648] HybridClock initialized: now 1790544614155818 us; error 48 us; skew 500 ppm
I20260927 21:30:14.156198 15705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:30:14 dist-test-slave-bxm0 krb5kdc[14257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:30:14.159634 15705 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 21:30:14.161219 15705 webserver.cc:460] Webserver started at http://127.0.97.3:44509/ using document root <none> and password file <none>
I20260927 21:30:14.162042 15705 fs_manager.cc:359] Metadata directory not provided
I20260927 21:30:14.162175 15705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:30:14.167676 15705 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:30:14.170522 15720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:14.172071 15705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260927 21:30:14.172346 15705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "b0ae63d02c9345f996a817c27132fb72"
format_stamp: "Formatted at 2026-09-27 21:30:08 on dist-test-slave-bxm0"
server_key: "8842a7693551bcd9dabad6498a38885a"
server_key_iv: "b2e8fa57a55481976f477e6953432d67"
server_key_version: "encryptionkey@0"
I20260927 21:30:14.173485 15705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:30:14.192476 15705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:30:14.197587 15705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:30:14.197996 15705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:30:14.199172 15705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:30:14.200727 15705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:30:14.200877 15705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:14.201049 15705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:30:14.201160 15705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:14.235579 15705 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41909
I20260927 21:30:14.235626 15832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41909 every 8 connection(s)
I20260927 21:30:14.237103 15705 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:30:14.242043 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:14.240171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.3:38315)
0927 21:30:14.240916 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:14.241102 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":294,"threads_started":1}
W20260927 21:30:14.242727 15833 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37797 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37797: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
I20260927 21:30:14.243570 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:14.243264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.3:43677)
0927 21:30:14.243403 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:14.243469 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 21:30:14.244336 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 15705
I20260927 21:30:14.244580 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:14.244231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.3:39727)
0927 21:30:14.244364 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:14.244433 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 21:30:14.244827 15833 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:30:14.319947 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:14.319227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.1:44569)
0927 21:30:14.319636 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:14.319723 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":196,"threads_started":1}
I20260927 21:30:14.493490 15838 raft_consensus.cc:482] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:30:14.493810 15838 raft_consensus.cc:504] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 } } peers { permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } } peers { permanent_uuid: "b0ae63d02c9345f996a817c27132fb72" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41909 } }
I20260927 21:30:14.495420 15838 leader_election.cc:288] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bfe26c8bab6e40c49bcde52dca387731 (127.0.97.2:41425), b0ae63d02c9345f996a817c27132fb72 (127.0.97.3:41909)
Sep 27 21:30:14 dist-test-slave-bxm0 krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
Sep 27 21:30:14 dist-test-slave-bxm0 krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 21:30:14.519682 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:14.495951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42119 (local address 127.0.97.2:41425)
0927 21:30:14.497565 (+ 1614us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:14.497585 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:14.497650 (+ 65us) server_negotiation.cc:407] Connection header received
0927 21:30:14.500055 (+ 2405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:14.500065 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:14.500434 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:14.500750 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:14.502693 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:14.504095 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:14.506811 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:14.507524 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:14.507756 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:14.512092 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:14.512099 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:14.512112 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:14.512151 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:14.515808 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:14.516553 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:14.516561 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:14.516571 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:14.516742 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:14.517165 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:14.517170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:14.517173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:14.517811 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:14.518017 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:14.518880 (+ 863us) server_negotiation.cc:299] Negotiation successful
0927 21:30:14.519254 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1376,"thread_start_us":1253,"threads_started":1}
I20260927 21:30:14.519683 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:14.496121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33735 (local address 127.0.97.3:41909)
0927 21:30:14.498309 (+ 2188us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:14.498324 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:14.498368 (+ 44us) server_negotiation.cc:407] Connection header received
0927 21:30:14.500076 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:14.500085 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:14.500428 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:14.500767 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:14.502700 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:14.504058 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:14.506846 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:14.507507 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:14.507720 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:14.512004 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:14.512013 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:14.512029 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:14.512067 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:14.515620 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:14.516505 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:14.516514 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:14.516522 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:14.516692 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:14.517178 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:14.517183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:14.517187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:14.517797 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:14.518059 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:14.518872 (+ 813us) server_negotiation.cc:299] Negotiation successful
0927 21:30:14.519272 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1994,"thread_start_us":1827,"threads_started":1}
I20260927 21:30:14.519680 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:14.495682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41425 (local address 127.0.97.1:42119)
0927 21:30:14.496016 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:14.496060 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:14.496708 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:14.500873 (+ 4165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:14.500885 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:14.501776 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:30:14.502502 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:14.502522 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:14.504299 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:14.504307 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:14.506623 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:14.506641 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:14.507776 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:14.507783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:14.508138 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:14.508942 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:30:14.509002 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:30:14.511776 (+ 2774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:30:14.515963 (+ 4187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:14.515972 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:14.515975 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:14.516403 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:14.516891 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:14.516895 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:14.516898 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:14.517045 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:14.517968 (+ 923us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:30:14.517976 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:30:14.518334 (+ 358us) client_negotiation.cc:769] Sending connection context
0927 21:30:14.518631 (+ 297us) client_negotiation.cc:240] Negotiation successful
0927 21:30:14.519194 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":319,"spinlock_wait_cycles":22528}
I20260927 21:30:14.520344 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:14.495775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41909 (local address 127.0.97.1:33735)
0927 21:30:14.497979 (+ 2204us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:14.497999 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:14.498117 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:14.500874 (+ 2757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:14.500885 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:14.501777 (+ 892us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:30:14.502501 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:14.502522 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:14.504277 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:14.504284 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:14.506623 (+ 2339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:14.506641 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:14.507748 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:14.507755 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:14.508133 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:14.508989 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:30:14.509012 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:30:14.511647 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:30:14.515826 (+ 4179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:14.515833 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:14.515847 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:14.516340 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:14.516818 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:14.516823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:14.516828 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:14.517024 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:14.517984 (+ 960us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:30:14.517989 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:30:14.518332 (+ 343us) client_negotiation.cc:769] Sending connection context
0927 21:30:14.518631 (+ 299us) client_negotiation.cc:240] Negotiation successful
0927 21:30:14.519350 (+ 719us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2023,"mutex_wait_us":474,"spinlock_wait_cycles":17792,"thread_start_us":1823,"threads_started":1}
W20260927 21:30:14.523102 15458 leader_election.cc:341] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0ae63d02c9345f996a817c27132fb72 (127.0.97.3:41909): Not found: Tablet not found: 86864a1aa8894b2f9524d81c89e637d3
W20260927 21:30:14.523419 15457 leader_election.cc:341] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bfe26c8bab6e40c49bcde52dca387731 (127.0.97.2:41425): Not found: Tablet not found: 86864a1aa8894b2f9524d81c89e637d3
I20260927 21:30:14.523557 15457 leader_election.cc:302] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f92ecd52d3014aeabccda938a17e7e68; no voters: b0ae63d02c9345f996a817c27132fb72, bfe26c8bab6e40c49bcde52dca387731
I20260927 21:30:14.523969 15838 raft_consensus.cc:2726] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:30:14.779093 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:14.778051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.2:51327)
0927 21:30:14.778684 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:14.778819 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":249,"threads_started":1}
I20260927 21:30:15.246138 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:15.245333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.3:49741)
0927 21:30:15.245823 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:15.245947 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":223,"threads_started":1}
I20260927 21:30:15.321583 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:15.320923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.1:57011)
0927 21:30:15.321353 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:15.321432 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":209,"threads_started":1}
I20260927 21:30:15.780628 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:15.779945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.2:51463)
0927 21:30:15.780392 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:15.780472 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":240,"threads_started":1}
I20260927 21:30:16.247768 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:16.247047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.3:47633)
0927 21:30:16.247495 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:16.247579 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":226,"threads_started":1}
I20260927 21:30:16.323330 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:16.322555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.1:40907)
0927 21:30:16.323048 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:16.323142 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":239,"threads_started":1}
I20260927 21:30:16.653860 15848 raft_consensus.cc:482] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:30:16.654078 15848 raft_consensus.cc:504] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 } } peers { permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } } peers { permanent_uuid: "b0ae63d02c9345f996a817c27132fb72" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41909 } }
I20260927 21:30:16.654757 15848 leader_election.cc:288] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bfe26c8bab6e40c49bcde52dca387731 (127.0.97.2:41425), b0ae63d02c9345f996a817c27132fb72 (127.0.97.3:41909)
W20260927 21:30:16.655784 15457 leader_election.cc:341] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bfe26c8bab6e40c49bcde52dca387731 (127.0.97.2:41425): Not found: Tablet not found: 86864a1aa8894b2f9524d81c89e637d3
W20260927 21:30:16.655951 15458 leader_election.cc:341] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0ae63d02c9345f996a817c27132fb72 (127.0.97.3:41909): Not found: Tablet not found: 86864a1aa8894b2f9524d81c89e637d3
I20260927 21:30:16.656093 15458 leader_election.cc:302] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f92ecd52d3014aeabccda938a17e7e68; no voters: b0ae63d02c9345f996a817c27132fb72, bfe26c8bab6e40c49bcde52dca387731
I20260927 21:30:16.656363 15848 raft_consensus.cc:2726] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:30:16.782344 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:16.781585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.2:60379)
0927 21:30:16.782094 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:16.782180 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":220,"threads_started":1}
I20260927 21:30:17.249281 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:17.248629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.3:60501)
0927 21:30:17.249045 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:17.249124 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":218,"threads_started":1}
I20260927 21:30:17.325179 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:17.324316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.1:39785)
0927 21:30:17.324871 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:17.324958 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":299,"threads_started":1}
I20260927 21:30:17.784219 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:17.783297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.2:51599)
0927 21:30:17.783829 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:17.783932 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":256,"threads_started":1}
I20260927 21:30:18.250947 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:18.250250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.3:59591)
0927 21:30:18.250696 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:18.250784 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":223,"threads_started":1}
I20260927 21:30:18.326957 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:18.326229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.1:60005)
0927 21:30:18.326698 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:18.326784 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":232,"threads_started":1}
I20260927 21:30:18.786173 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:18.785353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.2:43883)
0927 21:30:18.785926 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:18.786013 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":361,"threads_started":1}
I20260927 21:30:19.252642 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:19.251977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.3:42945)
0927 21:30:19.252399 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:19.252478 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":221,"threads_started":1}
I20260927 21:30:19.328701 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:19.327992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.1:37333)
0927 21:30:19.328453 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:19.328546 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":225,"threads_started":1}
I20260927 21:30:19.787875 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:19.787104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.2:55253)
0927 21:30:19.787607 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:19.787693 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":285,"threads_started":1}
I20260927 21:30:19.910897 15859 raft_consensus.cc:482] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:30:19.911146 15859 raft_consensus.cc:504] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 } } peers { permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } } peers { permanent_uuid: "b0ae63d02c9345f996a817c27132fb72" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41909 } }
I20260927 21:30:19.911839 15859 leader_election.cc:288] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bfe26c8bab6e40c49bcde52dca387731 (127.0.97.2:41425), b0ae63d02c9345f996a817c27132fb72 (127.0.97.3:41909)
W20260927 21:30:19.912782 15458 leader_election.cc:341] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0ae63d02c9345f996a817c27132fb72 (127.0.97.3:41909): Not found: Tablet not found: 86864a1aa8894b2f9524d81c89e637d3
W20260927 21:30:19.912949 15457 leader_election.cc:341] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bfe26c8bab6e40c49bcde52dca387731 (127.0.97.2:41425): Not found: Tablet not found: 86864a1aa8894b2f9524d81c89e637d3
I20260927 21:30:19.913127 15457 leader_election.cc:302] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f92ecd52d3014aeabccda938a17e7e68; no voters: b0ae63d02c9345f996a817c27132fb72, bfe26c8bab6e40c49bcde52dca387731
I20260927 21:30:19.913401 15859 raft_consensus.cc:2726] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:30:20.254403 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:20.253596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.3:55201)
0927 21:30:20.254120 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:20.254205 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":232,"threads_started":1}
I20260927 21:30:20.330459 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:20.329699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.1:34003)
0927 21:30:20.330161 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:20.330255 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":230,"threads_started":1}
I20260927 21:30:20.789870 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:20.789006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.2:58983)
0927 21:30:20.789531 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:20.789657 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":225,"threads_started":1}
I20260927 21:30:21.256079 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:21.255267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.3:34735)
0927 21:30:21.255765 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:21.255880 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":243,"threads_started":1}
I20260927 21:30:21.332031 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:21.331305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.1:50735)
0927 21:30:21.331764 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:21.331854 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":256,"threads_started":1}
I20260927 21:30:21.791867 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:21.790965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.2:34843)
0927 21:30:21.791507 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:21.791611 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":265,"threads_started":1}
I20260927 21:30:22.257871 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:22.257064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.3:48985)
0927 21:30:22.257504 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:22.257667 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":235,"threads_started":1}
I20260927 21:30:22.333811 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:22.332975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.1:58129)
0927 21:30:22.333457 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:22.333567 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":242,"threads_started":1}
I20260927 21:30:22.794070 15868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:22.793128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.2:40133)
0927 21:30:22.793720 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:22.793862 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":281,"threads_started":1}
I20260927 21:30:23.259562 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:23.258865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.3:54783)
0927 21:30:23.259323 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:23.259410 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":221,"threads_started":1}
I20260927 21:30:23.335529 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:23.334849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.1:42113)
0927 21:30:23.335260 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:23.335338 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":212,"threads_started":1}
I20260927 21:30:23.795753 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:23.795060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.2:55601)
0927 21:30:23.795487 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:23.795568 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":230,"threads_started":1}
I20260927 21:30:24.261240 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:24.260547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.3:54483)
0927 21:30:24.260978 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:24.261061 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":227,"threads_started":1}
I20260927 21:30:24.337105 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:24.336416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.1:34271)
0927 21:30:24.336845 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:24.336932 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":215,"threads_started":1}
I20260927 21:30:24.797540 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:24.796735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.2:49579)
0927 21:30:24.797271 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:24.797383 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":269,"threads_started":1}
I20260927 21:30:24.948675 15875 raft_consensus.cc:482] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:30:24.948889 15875 raft_consensus.cc:504] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f92ecd52d3014aeabccda938a17e7e68" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45047 } } peers { permanent_uuid: "bfe26c8bab6e40c49bcde52dca387731" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41425 } } peers { permanent_uuid: "b0ae63d02c9345f996a817c27132fb72" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41909 } }
I20260927 21:30:24.949582 15875 leader_election.cc:288] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bfe26c8bab6e40c49bcde52dca387731 (127.0.97.2:41425), b0ae63d02c9345f996a817c27132fb72 (127.0.97.3:41909)
W20260927 21:30:24.950496 15457 leader_election.cc:341] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bfe26c8bab6e40c49bcde52dca387731 (127.0.97.2:41425): Not found: Tablet not found: 86864a1aa8894b2f9524d81c89e637d3
W20260927 21:30:24.950671 15458 leader_election.cc:341] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0ae63d02c9345f996a817c27132fb72 (127.0.97.3:41909): Not found: Tablet not found: 86864a1aa8894b2f9524d81c89e637d3
I20260927 21:30:24.950745 15458 leader_election.cc:302] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f92ecd52d3014aeabccda938a17e7e68; no voters: b0ae63d02c9345f996a817c27132fb72, bfe26c8bab6e40c49bcde52dca387731
I20260927 21:30:24.950941 15875 raft_consensus.cc:2726] T 86864a1aa8894b2f9524d81c89e637d3 P f92ecd52d3014aeabccda938a17e7e68 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:30:25.262992 15876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:25.262270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.3:55783)
0927 21:30:25.262753 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:25.262844 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":252,"threads_started":1}
I20260927 21:30:25.338896 15877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:25.338038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.1:34277)
0927 21:30:25.338590 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:25.338672 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":255,"threads_started":1}
I20260927 21:30:25.799544 15878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:25.798668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.2:44793)
0927 21:30:25.799213 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:25.799302 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":249,"threads_started":1}
I20260927 21:30:26.264567 15879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:26.263886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.3:48691)
0927 21:30:26.264312 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:26.264388 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":219,"threads_started":1}
I20260927 21:30:26.340842 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:26.339902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.1:36785)
0927 21:30:26.340527 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:26.340628 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":301,"threads_started":1}
I20260927 21:30:26.801357 15881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:26.800570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.2:50453)
0927 21:30:26.801088 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:26.801189 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":266,"threads_started":1}
I20260927 21:30:27.266358 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:27.265592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.3:33381)
0927 21:30:27.266122 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:27.266206 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":229,"threads_started":1}
I20260927 21:30:27.342871 15883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:27.341865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.1:57007)
0927 21:30:27.342522 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:27.342657 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":342,"threads_started":1}
I20260927 21:30:27.803298 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:27.802508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.2:54665)
0927 21:30:27.803001 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:27.803093 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":268,"threads_started":1}
I20260927 21:30:28.268306 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:28.267447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.3:41473)
0927 21:30:28.267930 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:28.268013 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":264,"threads_started":1}
I20260927 21:30:28.345029 15886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:28.344221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.1:37815)
0927 21:30:28.344748 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:28.344852 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":274,"threads_started":1}
I20260927 21:30:28.805421 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:28.804624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.2:35867)
0927 21:30:28.805124 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:28.805249 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":263,"threads_started":1}
I20260927 21:30:29.250715 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 15405
I20260927 21:30:29.270601 15889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:29.269704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.3:52119)
0927 21:30:29.270309 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:29.270420 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":285,"threads_started":1}
I20260927 21:30:29.275962 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:29.251901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56678 (local address 127.0.97.1:45047)
0927 21:30:29.252733 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:29.252741 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:29.252766 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:30:29.252868 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:29.252879 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:29.253251 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:29.253400 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:29.255329 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:29.256342 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:29.257588 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:29.258269 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:29.258376 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:29.269104 (+ 10728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:29.269125 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:29.269141 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:29.269181 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:29.272580 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:29.273824 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:29.273837 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:29.273839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:29.273986 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:29.274412 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:29.274415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:29.274417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:29.274815 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:29.275046 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:29.275340 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 21:30:29.275556 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":568,"threads_started":1}
I20260927 21:30:29.277400 15566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:30:29.368770 15891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:29.367596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37797 (local address 127.0.97.1:34689)
0927 21:30:29.368232 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:29.368404 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":296,"threads_started":1}
I20260927 21:30:29.368808 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 15405
I20260927 21:30:29.384663 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 15574
I20260927 21:30:29.399104 15892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:29.385565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41552 (local address 127.0.97.2:41425)
0927 21:30:29.386189 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:29.386194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:29.386211 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:30:29.386270 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:29.386275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:29.386342 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:29.386436 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:29.387776 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:29.388548 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:29.389916 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:29.390342 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:29.390439 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:29.393152 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:29.393156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:29.393159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:29.393192 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:29.396177 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:29.396850 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:29.396854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:29.396857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:29.397017 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:29.397384 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:29.397388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:29.397390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:29.397945 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:29.398173 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:29.398455 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 21:30:29.398641 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":354,"threads_started":1}
I20260927 21:30:29.399888 15697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:30:29.486402 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 15574
I20260927 21:30:29.504347 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 15705
I20260927 21:30:29.517973 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:29.505467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44260 (local address 127.0.97.3:41909)
0927 21:30:29.506029 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:29.506034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:29.506050 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:30:29.506103 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:29.506106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:29.506176 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:29.506291 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:29.507600 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:29.508408 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:29.509708 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:29.510153 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:29.510261 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:29.512860 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:29.512864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:29.512867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:29.512900 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:29.515628 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:29.516300 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:29.516304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:29.516306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:29.516430 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:29.516744 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:29.516746 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:29.516748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:29.516999 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:29.517188 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:29.517445 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 21:30:29.517661 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":307,"threads_started":1}
I20260927 21:30:29.518674 15828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:30:29.627626 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 15705
2026-09-27T21:30:29Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24511 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:30:29 dist-test-slave-bxm0 krb5kdc[15898](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:30:29 dist-test-slave-bxm0 krb5kdc[15898](info): set up 2 sockets
Sep 27 21:30:29 dist-test-slave-bxm0 krb5kdc[15898](info): commencing operation
krb5kdc: starting...
W20260927 21:30:30.738323 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:30:30 dist-test-slave-bxm0 krb5kdc[15898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:30:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:30:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:30:30.991773 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46129
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:30:31.338300 15911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:30:31.338708 15911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:30:31.338907 15911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:30:31.346630 15911 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 21:30:31.346819 15911 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 21:30:31.346903 15911 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 21:30:31.346992 15911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:30:31.347126 15911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:30:31.347266 15911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:30:31.347368 15911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:30:31.347458 15911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:30:31.358085 15911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:30:31.359545 15911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:30:31.361021 15911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:30:31.373656 15916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:31.373690 15919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:31.373579 15917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:30:31.374562 15911 server_base.cc:1034] running on GCE node
I20260927 21:30:31.375427 15911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:30:31.376914 15911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:30:31.378151 15911 hybrid_clock.cc:648] HybridClock initialized: now 1790544631378077 us; error 56 us; skew 500 ppm
I20260927 21:30:31.378484 15911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:30:31 dist-test-slave-bxm0 krb5kdc[15898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:30:31.381603 15911 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:30:31.383188 15911 webserver.cc:460] Webserver started at http://127.0.97.62:42619/ using document root <none> and password file <none>
I20260927 21:30:31.383919 15911 fs_manager.cc:359] Metadata directory not provided
I20260927 21:30:31.384047 15911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:30:31.384368 15911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:30:31.387506 15911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "218b963fcfda487396ccc45be614fc68"
format_stamp: "Formatted at 2026-09-27 21:30:31 on dist-test-slave-bxm0"
server_key: "51ddf1bd8e2eebf77acb479f6f8d1579"
server_key_iv: "61e116dda024c40246d62d028d1bae0b"
server_key_version: "encryptionkey@0"
I20260927 21:30:31.388302 15911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "218b963fcfda487396ccc45be614fc68"
format_stamp: "Formatted at 2026-09-27 21:30:31 on dist-test-slave-bxm0"
server_key: "51ddf1bd8e2eebf77acb479f6f8d1579"
server_key_iv: "61e116dda024c40246d62d028d1bae0b"
server_key_version: "encryptionkey@0"
I20260927 21:30:31.393780 15911 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 21:30:31.397413 15926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:31.398813 15911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:30:31.399001 15911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "218b963fcfda487396ccc45be614fc68"
format_stamp: "Formatted at 2026-09-27 21:30:31 on dist-test-slave-bxm0"
server_key: "51ddf1bd8e2eebf77acb479f6f8d1579"
server_key_iv: "61e116dda024c40246d62d028d1bae0b"
server_key_version: "encryptionkey@0"
I20260927 21:30:31.399222 15911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:30:31.421473 15911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:30:31.426242 15911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:30:31.426559 15911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:30:31.445425 15911 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46129
I20260927 21:30:31.445459 15977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46129 every 8 connection(s)
I20260927 21:30:31.446884 15911 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:30:31.447419 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 15911
I20260927 21:30:31.447669 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:30:31.448536 388 external_mini_cluster.cc:1428] Setting key 7bf7db97a404c1dd50e16db545a73f53
I20260927 21:30:31.451814 15978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 21:30:31 dist-test-slave-bxm0 krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:30:31.463052 15978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68: Bootstrap starting.
I20260927 21:30:31.466638 15978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68: Neither blocks nor log segments found. Creating new log.
I20260927 21:30:31.467949 15978 log.cc:824] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68: Log is configured to *not* fsync() on all Append() calls
I20260927 21:30:31.471567 15978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68: No bootstrap required, opened a new log
I20260927 21:30:31.471971 15981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:31.450786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43804 (local address 127.0.97.62:46129)
0927 21:30:31.451545 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:31.451558 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:31.451597 (+ 39us) server_negotiation.cc:407] Connection header received
0927 21:30:31.453304 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:31.453326 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:31.453772 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:31.454175 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:31.455650 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:31.456995 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:31.458501 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:31.459197 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:31.459435 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:31.463915 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:31.463924 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:31.463940 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:31.463981 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:31.467187 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:31.468216 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:31.468223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:31.468230 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:31.468434 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:31.468960 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:31.468968 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:31.468974 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:31.469751 (+ 777us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:31.469987 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:31.470460 (+ 473us) server_negotiation.cc:299] Negotiation successful
0927 21:30:31.470872 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":336,"threads_started":1}
I20260927 21:30:31.479542 15978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218b963fcfda487396ccc45be614fc68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46129 } }
I20260927 21:30:31.479959 15978 raft_consensus.cc:374] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:30:31.480082 15978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218b963fcfda487396ccc45be614fc68, State: Initialized, Role: FOLLOWER
I20260927 21:30:31.480769 15978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218b963fcfda487396ccc45be614fc68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46129 } }
I20260927 21:30:31.481005 15978 raft_consensus.cc:388] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:30:31.481140 15978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:30:31.481320 15978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:30:31.483757 15978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218b963fcfda487396ccc45be614fc68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46129 } }
I20260927 21:30:31.484385 15978 leader_election.cc:302] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 218b963fcfda487396ccc45be614fc68; no voters:
I20260927 21:30:31.484998 15978 leader_election.cc:288] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:30:31.485144 15983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:30:31.485869 15983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [term 1 LEADER]: Becoming Leader. State: Replica: 218b963fcfda487396ccc45be614fc68, State: Running, Role: LEADER
I20260927 21:30:31.486449 15983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218b963fcfda487396ccc45be614fc68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46129 } }
I20260927 21:30:31.487252 15978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:30:31.489535 15985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 218b963fcfda487396ccc45be614fc68. Latest consensus state: current_term: 1 leader_uuid: "218b963fcfda487396ccc45be614fc68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218b963fcfda487396ccc45be614fc68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46129 } } }
I20260927 21:30:31.490090 15985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [sys.catalog]: This master's current role is: LEADER
I20260927 21:30:31.489537 15984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "218b963fcfda487396ccc45be614fc68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218b963fcfda487396ccc45be614fc68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46129 } } }
I20260927 21:30:31.490526 15984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [sys.catalog]: This master's current role is: LEADER
I20260927 21:30:31.492559 15992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:30:31.497980 15992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:30:31.508031 15992 catalog_manager.cc:1347] Generated new cluster ID: 5840b2607efb448692ec1693ad1f32f4
I20260927 21:30:31.508229 15992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:30:31.525071 15992 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:30:31.527140 15992 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:30:31.545998 15992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68: Generated new TSK 0
I20260927 21:30:31.547183 15992 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 21:30:31.614815 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46129
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:30:31.958578 16004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:30:31.958946 16004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:30:31.959127 16004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:30:31.966607 16004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:30:31.966790 16004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:30:31.966996 16004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:30:31.977551 16004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:30:31.978974 16004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:30:31.980346 16004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:30:31.994666 16009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:31.994700 16010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:31.994726 16012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:30:31.995532 16004 server_base.cc:1034] running on GCE node
I20260927 21:30:31.996205 16004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:30:31.997200 16004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:30:31.998467 16004 hybrid_clock.cc:648] HybridClock initialized: now 1790544631998420 us; error 41 us; skew 500 ppm
I20260927 21:30:31.998827 16004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:30:31 dist-test-slave-bxm0 krb5kdc[15898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:30:32.002044 16004 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 21:30:32.003696 16004 webserver.cc:460] Webserver started at http://127.0.97.1:41649/ using document root <none> and password file <none>
I20260927 21:30:32.004458 16004 fs_manager.cc:359] Metadata directory not provided
I20260927 21:30:32.004602 16004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:30:32.004961 16004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:30:32.008179 16004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/data/instance:
uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4"
format_stamp: "Formatted at 2026-09-27 21:30:31 on dist-test-slave-bxm0"
server_key: "ee68e6ab7ef8d32856e58ac1cb492159"
server_key_iv: "1f464dc9651e30be35e55e8bc50e397f"
server_key_version: "encryptionkey@0"
I20260927 21:30:32.009039 16004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4"
format_stamp: "Formatted at 2026-09-27 21:30:31 on dist-test-slave-bxm0"
server_key: "ee68e6ab7ef8d32856e58ac1cb492159"
server_key_iv: "1f464dc9651e30be35e55e8bc50e397f"
server_key_version: "encryptionkey@0"
I20260927 21:30:32.014585 16004 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260927 21:30:32.018499 16019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:32.019980 16004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 21:30:32.020166 16004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4"
format_stamp: "Formatted at 2026-09-27 21:30:31 on dist-test-slave-bxm0"
server_key: "ee68e6ab7ef8d32856e58ac1cb492159"
server_key_iv: "1f464dc9651e30be35e55e8bc50e397f"
server_key_version: "encryptionkey@0"
I20260927 21:30:32.020390 16004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:30:32.037158 16004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:30:32.041720 16004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:30:32.042052 16004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:30:32.043107 16004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:30:32.044512 16004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:30:32.044624 16004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:32.044788 16004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:30:32.044889 16004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:32.077570 16004 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37393
I20260927 21:30:32.077631 16131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37393 every 8 connection(s)
I20260927 21:30:32.079036 16004 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:30:32.084324 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 16004
I20260927 21:30:32.084556 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/wal/instance
I20260927 21:30:32.085397 388 external_mini_cluster.cc:1428] Setting key c442cc8154d2f9027ccfa0ebe1630b73
Sep 27 21:30:32 dist-test-slave-bxm0 krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:30:32.104820 16134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:32.081951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46095 (local address 127.0.97.62:46129)
0927 21:30:32.082421 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:32.082426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:32.083467 (+ 1041us) server_negotiation.cc:407] Connection header received
0927 21:30:32.086237 (+ 2770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:32.086243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:32.086346 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:32.086472 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:32.088565 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:32.089303 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:32.091985 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:32.092461 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:32.092559 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:32.097132 (+ 4573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:32.097141 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:32.097148 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:32.097202 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:32.100867 (+ 3665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:32.101822 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:32.101831 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:32.101838 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:32.102002 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:32.102678 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:32.102686 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:32.102692 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:32.103089 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:32.103255 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:32.104031 (+ 776us) server_negotiation.cc:299] Negotiation successful
0927 21:30:32.104325 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":279,"threads_started":1}
I20260927 21:30:32.105439 16135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:32.082491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.1:46095)
0927 21:30:32.083277 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:32.083323 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:32.085882 (+ 2559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:32.086742 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:32.086751 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:32.087413 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:30:32.088345 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:32.088377 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:32.089496 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:32.089505 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:32.091810 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:32.091830 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:32.092707 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:32.092717 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:32.093077 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:32.093958 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:30:32.093994 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:30:32.096850 (+ 2856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:30:32.101065 (+ 4215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:32.101074 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:32.101089 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:32.101593 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:32.102163 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:32.102168 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:32.102171 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:32.102543 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:32.103246 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:30:32.103253 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:30:32.103618 (+ 365us) client_negotiation.cc:769] Sending connection context
0927 21:30:32.103862 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 21:30:32.104305 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":302,"threads_started":1}
I20260927 21:30:32.107097 16132 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46129
I20260927 21:30:32.107445 16132 heartbeater.cc:464] Registering TS with master...
I20260927 21:30:32.108235 16132 heartbeater.cc:511] Master 127.0.97.62:46129 requested a full tablet report, sending...
I20260927 21:30:32.110664 15943 ts_manager.cc:194] Registered new tserver with Master: 6bdafe2bbc0c430f9bdd4e714ef4efa4 (127.0.97.1:37393)
I20260927 21:30:32.112105 15943 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46095
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 21:30:32.144860 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46129
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:30:32.478250 16138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:30:32.478543 16138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:30:32.478638 16138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:30:32.485982 16138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:30:32.486106 16138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:30:32.486227 16138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:30:32.496673 16138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:30:32.498075 16138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:30:32.499346 16138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:30:32.513187 16146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:32.513187 16143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:32.513291 16144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:30:32.514117 16138 server_base.cc:1034] running on GCE node
I20260927 21:30:32.514704 16138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:30:32.515651 16138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:30:32.516853 16138 hybrid_clock.cc:648] HybridClock initialized: now 1790544632516811 us; error 29 us; skew 500 ppm
I20260927 21:30:32.517194 16138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:30:32 dist-test-slave-bxm0 krb5kdc[15898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:30:32.520397 16138 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 21:30:32.522087 16138 webserver.cc:460] Webserver started at http://127.0.97.2:33923/ using document root <none> and password file <none>
I20260927 21:30:32.522817 16138 fs_manager.cc:359] Metadata directory not provided
I20260927 21:30:32.522969 16138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:30:32.523305 16138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:30:32.526487 16138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/data/instance:
uuid: "60cef69369a34800bf13b929dc656f86"
format_stamp: "Formatted at 2026-09-27 21:30:32 on dist-test-slave-bxm0"
server_key: "8b5fa549719d4727a8e15d8ced4b2d45"
server_key_iv: "855ee0abffeceb8073da773360a377b5"
server_key_version: "encryptionkey@0"
I20260927 21:30:32.527278 16138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/wal/instance:
uuid: "60cef69369a34800bf13b929dc656f86"
format_stamp: "Formatted at 2026-09-27 21:30:32 on dist-test-slave-bxm0"
server_key: "8b5fa549719d4727a8e15d8ced4b2d45"
server_key_iv: "855ee0abffeceb8073da773360a377b5"
server_key_version: "encryptionkey@0"
I20260927 21:30:32.532707 16138 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 21:30:32.536623 16153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:32.538179 16138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 21:30:32.538407 16138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "60cef69369a34800bf13b929dc656f86"
format_stamp: "Formatted at 2026-09-27 21:30:32 on dist-test-slave-bxm0"
server_key: "8b5fa549719d4727a8e15d8ced4b2d45"
server_key_iv: "855ee0abffeceb8073da773360a377b5"
server_key_version: "encryptionkey@0"
I20260927 21:30:32.538647 16138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:30:32.555797 16138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:30:32.560187 16138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:30:32.560534 16138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:30:32.561589 16138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:30:32.563117 16138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:30:32.563235 16138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:32.563390 16138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:30:32.563462 16138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:32.598795 16138 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44853
I20260927 21:30:32.598896 16265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44853 every 8 connection(s)
I20260927 21:30:32.600152 16138 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:30:32.609428 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 16138
I20260927 21:30:32.609596 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/wal/instance
I20260927 21:30:32.610286 388 external_mini_cluster.cc:1428] Setting key a1758f635bb76d0d82cb77a6c761076f
Sep 27 21:30:32 dist-test-slave-bxm0 krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:30:32.622929 16134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:32.602942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55581 (local address 127.0.97.62:46129)
0927 21:30:32.603170 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:32.603177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:32.604595 (+ 1418us) server_negotiation.cc:407] Connection header received
0927 21:30:32.606755 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:32.606762 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:32.606873 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:32.607002 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:32.608742 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:32.609696 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:32.611970 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:32.612580 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:32.612723 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:32.616756 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:32.616761 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:32.616764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:32.616799 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:32.619511 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:32.620351 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:32.620360 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:32.620365 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:32.620505 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:32.620960 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:32.620967 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:32.620972 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:32.621326 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:32.621486 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:32.622314 (+ 828us) server_negotiation.cc:299] Negotiation successful
0927 21:30:32.622555 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 21:30:32.624045 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:32.603500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.2:55581)
0927 21:30:32.604404 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:32.604459 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:32.606426 (+ 1967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:32.607270 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:32.607279 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:32.607862 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:30:32.608480 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:32.608499 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:32.609915 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:32.609921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:32.611788 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:32.611804 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:32.612842 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:32.612851 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:32.613102 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:32.613886 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:30:32.613916 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:30:32.616471 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:30:32.619682 (+ 3211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:32.619692 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:32.619712 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:32.620212 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:32.620623 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:32.620628 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:32.620632 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:32.620833 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:32.621460 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:30:32.621468 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:30:32.621915 (+ 447us) client_negotiation.cc:769] Sending connection context
0927 21:30:32.622210 (+ 295us) client_negotiation.cc:240] Negotiation successful
0927 21:30:32.622774 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":399,"threads_started":1}
I20260927 21:30:32.625931 16266 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46129
I20260927 21:30:32.626273 16266 heartbeater.cc:464] Registering TS with master...
I20260927 21:30:32.627038 16266 heartbeater.cc:511] Master 127.0.97.62:46129 requested a full tablet report, sending...
I20260927 21:30:32.628638 15943 ts_manager.cc:194] Registered new tserver with Master: 60cef69369a34800bf13b929dc656f86 (127.0.97.2:44853)
I20260927 21:30:32.629474 15943 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55581
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 21:30:32.666060 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46129
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:30:33.017023 16271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:30:33.017366 16271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:30:33.017470 16271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:30:33.025583 16271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:30:33.025789 16271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:30:33.025923 16271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:30:33.036482 16271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:30:33.037976 16271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:30:33.039361 16271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:30:33.053474 16276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:33.053388 16279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:33.053408 16277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:30:33.054636 16271 server_base.cc:1034] running on GCE node
I20260927 21:30:33.055266 16271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:30:33.056272 16271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:30:33.057507 16271 hybrid_clock.cc:648] HybridClock initialized: now 1790544633057452 us; error 52 us; skew 500 ppm
I20260927 21:30:33.057869 16271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:30:33 dist-test-slave-bxm0 krb5kdc[15898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:30:33.061107 16271 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 21:30:33.062656 16271 webserver.cc:460] Webserver started at http://127.0.97.3:33597/ using document root <none> and password file <none>
I20260927 21:30:33.063344 16271 fs_manager.cc:359] Metadata directory not provided
I20260927 21:30:33.063446 16271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:30:33.063710 16271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:30:33.066761 16271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/data/instance:
uuid: "46b4f74760a6467ea467ea21c10205b4"
format_stamp: "Formatted at 2026-09-27 21:30:33 on dist-test-slave-bxm0"
server_key: "388ac51b07723c9373b0bd6e567650bc"
server_key_iv: "93fe6426bd6bec64db66ca76634e76d4"
server_key_version: "encryptionkey@0"
I20260927 21:30:33.067477 16271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/wal/instance:
uuid: "46b4f74760a6467ea467ea21c10205b4"
format_stamp: "Formatted at 2026-09-27 21:30:33 on dist-test-slave-bxm0"
server_key: "388ac51b07723c9373b0bd6e567650bc"
server_key_iv: "93fe6426bd6bec64db66ca76634e76d4"
server_key_version: "encryptionkey@0"
I20260927 21:30:33.072903 16271 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 21:30:33.076783 16286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:33.078671 16271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:30:33.078866 16271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "46b4f74760a6467ea467ea21c10205b4"
format_stamp: "Formatted at 2026-09-27 21:30:33 on dist-test-slave-bxm0"
server_key: "388ac51b07723c9373b0bd6e567650bc"
server_key_iv: "93fe6426bd6bec64db66ca76634e76d4"
server_key_version: "encryptionkey@0"
I20260927 21:30:33.079101 16271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:30:33.097880 16271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:30:33.102252 16271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:30:33.102569 16271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:30:33.103713 16271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:30:33.105127 16271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:30:33.105229 16271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:33.105381 16271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:30:33.105455 16271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:33.115411 16132 heartbeater.cc:503] Master 127.0.97.62:46129 was elected leader, sending a full tablet report...
I20260927 21:30:33.139822 16271 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33005
I20260927 21:30:33.139866 16398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33005 every 8 connection(s)
I20260927 21:30:33.141119 16271 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:30:33.142072 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 16271
I20260927 21:30:33.142263 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/wal/instance
I20260927 21:30:33.142819 388 external_mini_cluster.cc:1428] Setting key 12a0ef312d5816b9599a97447c5c7a96
Sep 27 21:30:33 dist-test-slave-bxm0 krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:30:33.162237 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:33.144032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48333 (local address 127.0.97.62:46129)
0927 21:30:33.144531 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:33.144535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:33.145676 (+ 1141us) server_negotiation.cc:407] Connection header received
0927 21:30:33.147741 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:33.147745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:33.147812 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:33.147912 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:33.149568 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.150404 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.151945 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.152368 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.152476 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:33.155967 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:33.155972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:33.155975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:33.156020 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:33.159315 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:33.160086 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:33.160090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:33.160092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:33.160227 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:33.160610 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:33.160613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:33.160615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:33.160893 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:33.161008 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:33.161776 (+ 768us) server_negotiation.cc:299] Negotiation successful
0927 21:30:33.161953 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":291,"threads_started":1}
I20260927 21:30:33.163182 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:33.144681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.3:48333)
0927 21:30:33.145393 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:33.145435 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:33.147469 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:33.148131 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:33.148138 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:33.148755 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:30:33.149375 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:33.149393 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.150551 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.150555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:33.151787 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:33.151799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.152562 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.152567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:33.152765 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:33.153464 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:30:33.153486 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:30:33.155738 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:30:33.159482 (+ 3744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:33.159495 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:33.159516 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:33.159977 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:33.160324 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:33.160328 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:33.160330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:33.160513 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:33.161004 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:30:33.161010 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:30:33.161346 (+ 336us) client_negotiation.cc:769] Sending connection context
0927 21:30:33.161583 (+ 237us) client_negotiation.cc:240] Negotiation successful
0927 21:30:33.162048 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":506,"threads_started":1}
I20260927 21:30:33.164572 16399 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46129
I20260927 21:30:33.164904 16399 heartbeater.cc:464] Registering TS with master...
I20260927 21:30:33.165725 16399 heartbeater.cc:511] Master 127.0.97.62:46129 requested a full tablet report, sending...
I20260927 21:30:33.167186 15943 ts_manager.cc:194] Registered new tserver with Master: 46b4f74760a6467ea467ea21c10205b4 (127.0.97.3:33005)
I20260927 21:30:33.167976 15943 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48333
I20260927 21:30:33.172075 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:30:33.189790 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:33.176549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43810 (local address 127.0.97.62:46129)
0927 21:30:33.176707 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:33.176711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:33.177193 (+ 482us) server_negotiation.cc:407] Connection header received
0927 21:30:33.177348 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:33.177352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:33.177416 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:33.177501 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:33.179386 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.180109 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.181416 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.181912 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.182013 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:33.184580 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:33.184584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:33.184587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:33.184633 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:33.187286 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:33.187996 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:33.188000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:33.188003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:33.188119 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:33.188509 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:33.188512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:33.188514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:33.188831 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:33.189004 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:33.189253 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 21:30:33.189427 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:30:33.194084 15943 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:30:33.201102 15943 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:30:33.236747 16412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:33.213701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44853 (local address 127.0.0.1:33138)
0927 21:30:33.224785 (+ 11084us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:33.224838 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:33.225079 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:33.226061 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:33.226070 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:33.226130 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:30:33.226561 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:33.226578 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.229059 (+ 2481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.229068 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:33.232480 (+ 3412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:33.232508 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.235259 (+ 2751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.235265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:33.235586 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:33.235672 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 21:30:33.235831 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 21:30:33.236118 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10811,"mutex_wait_us":36,"thread_start_us":10710,"threads_started":1}
I20260927 21:30:33.238380 16414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:33.213942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37393 (local address 127.0.0.1:50144)
0927 21:30:33.226225 (+ 12283us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:33.226246 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:33.226382 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:33.227608 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:33.227616 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:33.227668 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:30:33.228222 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:33.228238 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.230836 (+ 2598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.230844 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:33.233996 (+ 3152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:33.234019 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.237082 (+ 3063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.237092 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:33.237378 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:33.237436 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 21:30:33.237550 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 21:30:33.237784 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12130,"mutex_wait_us":5,"thread_start_us":11963,"threads_started":1}
I20260927 21:30:33.239140 16413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:33.213808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33138 (local address 127.0.97.2:44853)
0927 21:30:33.224249 (+ 10441us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:33.224277 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:33.225245 (+ 968us) server_negotiation.cc:407] Connection header received
0927 21:30:33.225393 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:33.225404 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:33.225723 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:33.226048 (+ 325us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:30:33.226808 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.228855 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.232786 (+ 3931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.234959 (+ 2173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.238239 (+ 3280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:33.238533 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:33.238625 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 21:30:33.238818 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10265,"thread_start_us":10149,"threads_started":1}
I20260927 21:30:33.239758 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:33.214140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50144 (local address 127.0.97.1:37393)
0927 21:30:33.225825 (+ 11685us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:33.225837 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:33.226539 (+ 702us) server_negotiation.cc:407] Connection header received
0927 21:30:33.226649 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:33.226670 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:33.227064 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:33.227293 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:30:33.228457 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.230607 (+ 2150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.234249 (+ 3642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.236364 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.236532 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:33.236872 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:33.238946 (+ 2074us) server_negotiation.cc:299] Negotiation successful
0927 21:30:33.239198 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11492,"thread_start_us":11295,"threads_started":1}
I20260927 21:30:33.243357 16201 tablet_service.cc:1467] Processing CreateTablet for tablet c39106a8bb0146558ee0a3eb67422cae (DEFAULT_TABLE table=test-table [id=dcab75b44e974539bbd77ffcc35ecfbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:30:33.244863 16201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c39106a8bb0146558ee0a3eb67422cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:30:33.244858 16067 tablet_service.cc:1467] Processing CreateTablet for tablet c39106a8bb0146558ee0a3eb67422cae (DEFAULT_TABLE table=test-table [id=dcab75b44e974539bbd77ffcc35ecfbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:30:33.246291 16067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c39106a8bb0146558ee0a3eb67422cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:30:33.247367 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:33.226763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43106 (local address 127.0.97.3:33005)
0927 21:30:33.238982 (+ 12219us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:33.238992 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:33.239028 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:30:33.239132 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:33.239142 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:33.239467 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:33.240316 (+ 849us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:30:33.240600 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.242159 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.244604 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.246171 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.246325 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:33.246555 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:33.246774 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 21:30:33.246937 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12023,"thread_start_us":12838,"threads_started":1}
I20260927 21:30:33.248020 16412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:33.226682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33005 (local address 127.0.0.1:43106)
0927 21:30:33.237839 (+ 11157us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:33.237864 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:33.238010 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:33.239900 (+ 1890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:33.239904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:33.239933 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:30:33.240123 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:33.240134 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.242317 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.242322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:33.244266 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:33.244293 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.246357 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.246365 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:33.246558 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:33.246594 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 21:30:33.246715 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 21:30:33.247724 (+ 1009us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10996,"thread_start_us":13858,"threads_started":1}
I20260927 21:30:33.251986 16334 tablet_service.cc:1467] Processing CreateTablet for tablet c39106a8bb0146558ee0a3eb67422cae (DEFAULT_TABLE table=test-table [id=dcab75b44e974539bbd77ffcc35ecfbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:30:33.253481 16334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c39106a8bb0146558ee0a3eb67422cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:30:33.258062 16418 tablet_bootstrap.cc:492] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4: Bootstrap starting.
I20260927 21:30:33.260123 16419 tablet_bootstrap.cc:492] T c39106a8bb0146558ee0a3eb67422cae P 60cef69369a34800bf13b929dc656f86: Bootstrap starting.
I20260927 21:30:33.262045 16418 tablet_bootstrap.cc:654] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4: Neither blocks nor log segments found. Creating new log.
I20260927 21:30:33.263411 16418 log.cc:824] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4: Log is configured to *not* fsync() on all Append() calls
I20260927 21:30:33.263931 16419 tablet_bootstrap.cc:654] T c39106a8bb0146558ee0a3eb67422cae P 60cef69369a34800bf13b929dc656f86: Neither blocks nor log segments found. Creating new log.
I20260927 21:30:33.265133 16419 log.cc:824] T c39106a8bb0146558ee0a3eb67422cae P 60cef69369a34800bf13b929dc656f86: Log is configured to *not* fsync() on all Append() calls
I20260927 21:30:33.266597 16418 tablet_bootstrap.cc:492] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4: No bootstrap required, opened a new log
I20260927 21:30:33.266901 16418 ts_tablet_manager.cc:1397] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 21:30:33.268244 16419 tablet_bootstrap.cc:492] T c39106a8bb0146558ee0a3eb67422cae P 60cef69369a34800bf13b929dc656f86: No bootstrap required, opened a new log
I20260927 21:30:33.268514 16419 ts_tablet_manager.cc:1397] T c39106a8bb0146558ee0a3eb67422cae P 60cef69369a34800bf13b929dc656f86: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 21:30:33.268837 16422 tablet_bootstrap.cc:492] T c39106a8bb0146558ee0a3eb67422cae P 46b4f74760a6467ea467ea21c10205b4: Bootstrap starting.
I20260927 21:30:33.271699 16422 tablet_bootstrap.cc:654] T c39106a8bb0146558ee0a3eb67422cae P 46b4f74760a6467ea467ea21c10205b4: Neither blocks nor log segments found. Creating new log.
I20260927 21:30:33.272739 16422 log.cc:824] T c39106a8bb0146558ee0a3eb67422cae P 46b4f74760a6467ea467ea21c10205b4: Log is configured to *not* fsync() on all Append() calls
I20260927 21:30:33.276434 16422 tablet_bootstrap.cc:492] T c39106a8bb0146558ee0a3eb67422cae P 46b4f74760a6467ea467ea21c10205b4: No bootstrap required, opened a new log
I20260927 21:30:33.276749 16422 ts_tablet_manager.cc:1397] T c39106a8bb0146558ee0a3eb67422cae P 46b4f74760a6467ea467ea21c10205b4: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 21:30:33.278095 16418 raft_consensus.cc:348] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60cef69369a34800bf13b929dc656f86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44853 } } peers { permanent_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37393 } } peers { permanent_uuid: "46b4f74760a6467ea467ea21c10205b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33005 } }
I20260927 21:30:33.278568 16418 raft_consensus.cc:374] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:30:33.278723 16418 raft_consensus.cc:729] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bdafe2bbc0c430f9bdd4e714ef4efa4, State: Initialized, Role: FOLLOWER
I20260927 21:30:33.279413 16418 consensus_queue.cc:260] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60cef69369a34800bf13b929dc656f86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44853 } } peers { permanent_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37393 } } peers { permanent_uuid: "46b4f74760a6467ea467ea21c10205b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33005 } }
I20260927 21:30:33.279296 16419 raft_consensus.cc:348] T c39106a8bb0146558ee0a3eb67422cae P 60cef69369a34800bf13b929dc656f86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60cef69369a34800bf13b929dc656f86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44853 } } peers { permanent_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37393 } } peers { permanent_uuid: "46b4f74760a6467ea467ea21c10205b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33005 } }
I20260927 21:30:33.279744 16419 raft_consensus.cc:374] T c39106a8bb0146558ee0a3eb67422cae P 60cef69369a34800bf13b929dc656f86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:30:33.279853 16419 raft_consensus.cc:729] T c39106a8bb0146558ee0a3eb67422cae P 60cef69369a34800bf13b929dc656f86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60cef69369a34800bf13b929dc656f86, State: Initialized, Role: FOLLOWER
I20260927 21:30:33.281294 16418 ts_tablet_manager.cc:1428] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4: Time spent starting tablet: real 0.014s user 0.006s sys 0.007s
I20260927 21:30:33.281857 16419 consensus_queue.cc:260] T c39106a8bb0146558ee0a3eb67422cae P 60cef69369a34800bf13b929dc656f86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60cef69369a34800bf13b929dc656f86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44853 } } peers { permanent_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37393 } } peers { permanent_uuid: "46b4f74760a6467ea467ea21c10205b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33005 } }
I20260927 21:30:33.283087 16266 heartbeater.cc:503] Master 127.0.97.62:46129 was elected leader, sending a full tablet report...
I20260927 21:30:33.283782 16419 ts_tablet_manager.cc:1428] T c39106a8bb0146558ee0a3eb67422cae P 60cef69369a34800bf13b929dc656f86: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260927 21:30:33.287021 16422 raft_consensus.cc:348] T c39106a8bb0146558ee0a3eb67422cae P 46b4f74760a6467ea467ea21c10205b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60cef69369a34800bf13b929dc656f86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44853 } } peers { permanent_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37393 } } peers { permanent_uuid: "46b4f74760a6467ea467ea21c10205b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33005 } }
I20260927 21:30:33.287668 16422 raft_consensus.cc:374] T c39106a8bb0146558ee0a3eb67422cae P 46b4f74760a6467ea467ea21c10205b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:30:33.287791 16422 raft_consensus.cc:729] T c39106a8bb0146558ee0a3eb67422cae P 46b4f74760a6467ea467ea21c10205b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46b4f74760a6467ea467ea21c10205b4, State: Initialized, Role: FOLLOWER
I20260927 21:30:33.288427 16422 consensus_queue.cc:260] T c39106a8bb0146558ee0a3eb67422cae P 46b4f74760a6467ea467ea21c10205b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60cef69369a34800bf13b929dc656f86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44853 } } peers { permanent_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37393 } } peers { permanent_uuid: "46b4f74760a6467ea467ea21c10205b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33005 } }
I20260927 21:30:33.289692 16399 heartbeater.cc:503] Master 127.0.97.62:46129 was elected leader, sending a full tablet report...
I20260927 21:30:33.290390 16422 ts_tablet_manager.cc:1428] T c39106a8bb0146558ee0a3eb67422cae P 46b4f74760a6467ea467ea21c10205b4: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
W20260927 21:30:33.331907 16133 tablet.cc:2369] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:30:33.352218 16267 tablet.cc:2369] T c39106a8bb0146558ee0a3eb67422cae P 60cef69369a34800bf13b929dc656f86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:30:33.362568 16424 raft_consensus.cc:482] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:30:33.362888 16424 raft_consensus.cc:504] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60cef69369a34800bf13b929dc656f86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44853 } } peers { permanent_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37393 } } peers { permanent_uuid: "46b4f74760a6467ea467ea21c10205b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33005 } }
I20260927 21:30:33.364655 16424 leader_election.cc:288] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60cef69369a34800bf13b929dc656f86 (127.0.97.2:44853), 46b4f74760a6467ea467ea21c10205b4 (127.0.97.3:33005)
I20260927 21:30:33.372283 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:33.365108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36581 (local address 127.0.97.3:33005)
0927 21:30:33.365332 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:33.365341 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:33.366244 (+ 903us) server_negotiation.cc:407] Connection header received
0927 21:30:33.366420 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:33.366425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:33.366511 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:33.366599 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:30:33.367301 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.368154 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.370529 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.371307 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.371405 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:33.371549 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:33.371960 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 21:30:33.372093 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 21:30:33.372298 16413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:33.365158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34153 (local address 127.0.97.2:44853)
0927 21:30:33.365371 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:33.365379 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:33.366423 (+ 1044us) server_negotiation.cc:407] Connection header received
0927 21:30:33.366581 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:33.366587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:33.366664 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:33.366766 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:30:33.367421 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.368107 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.370564 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.371323 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.371484 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:33.371582 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:33.371914 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 21:30:33.372040 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 21:30:33.372540 16428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:33.365180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44853 (local address 127.0.97.1:34153)
0927 21:30:33.366253 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:33.366279 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:33.366437 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:33.366797 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:33.366800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:33.366875 (+ 75us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:30:33.367253 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:33.367273 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.368238 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.368245 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:33.370375 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:33.370391 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.371583 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.371590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:33.371740 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:33.371765 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 21:30:33.371886 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 21:30:33.372018 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"mutex_wait_us":117,"spinlock_wait_cycles":19200,"thread_start_us":511,"threads_started":1}
I20260927 21:30:33.373025 16427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:33.365136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33005 (local address 127.0.97.1:36581)
0927 21:30:33.366120 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:33.366136 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:33.366278 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:33.366627 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:33.366631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:33.366655 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:30:33.367144 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:33.367158 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.368303 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.368311 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:33.370376 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:33.370389 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.371538 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.371545 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:33.371731 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:33.371781 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 21:30:33.372505 (+ 724us) client_negotiation.cc:240] Negotiation successful
0927 21:30:33.372638 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":74,"thread_start_us":561,"threads_started":1}
I20260927 21:30:33.373711 16221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39106a8bb0146558ee0a3eb67422cae" candidate_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60cef69369a34800bf13b929dc656f86" is_pre_election: true
I20260927 21:30:33.374037 16354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39106a8bb0146558ee0a3eb67422cae" candidate_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46b4f74760a6467ea467ea21c10205b4" is_pre_election: true
I20260927 21:30:33.374243 16221 raft_consensus.cc:2454] T c39106a8bb0146558ee0a3eb67422cae P 60cef69369a34800bf13b929dc656f86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bdafe2bbc0c430f9bdd4e714ef4efa4 in term 0.
I20260927 21:30:33.374583 16354 raft_consensus.cc:2454] T c39106a8bb0146558ee0a3eb67422cae P 46b4f74760a6467ea467ea21c10205b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bdafe2bbc0c430f9bdd4e714ef4efa4 in term 0.
I20260927 21:30:33.375098 16020 leader_election.cc:302] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60cef69369a34800bf13b929dc656f86, 6bdafe2bbc0c430f9bdd4e714ef4efa4; no voters:
I20260927 21:30:33.375510 16424 raft_consensus.cc:2781] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:30:33.375662 16424 raft_consensus.cc:482] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:30:33.375764 16424 raft_consensus.cc:3037] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:30:33.378060 16424 raft_consensus.cc:504] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60cef69369a34800bf13b929dc656f86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44853 } } peers { permanent_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37393 } } peers { permanent_uuid: "46b4f74760a6467ea467ea21c10205b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33005 } }
I20260927 21:30:33.378705 16424 leader_election.cc:288] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [CANDIDATE]: Term 1 election: Requested vote from peers 60cef69369a34800bf13b929dc656f86 (127.0.97.2:44853), 46b4f74760a6467ea467ea21c10205b4 (127.0.97.3:33005)
I20260927 21:30:33.379231 16221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39106a8bb0146558ee0a3eb67422cae" candidate_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60cef69369a34800bf13b929dc656f86"
I20260927 21:30:33.379346 16354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39106a8bb0146558ee0a3eb67422cae" candidate_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46b4f74760a6467ea467ea21c10205b4"
I20260927 21:30:33.379455 16221 raft_consensus.cc:3037] T c39106a8bb0146558ee0a3eb67422cae P 60cef69369a34800bf13b929dc656f86 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:30:33.379593 16354 raft_consensus.cc:3037] T c39106a8bb0146558ee0a3eb67422cae P 46b4f74760a6467ea467ea21c10205b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:30:33.381752 16221 raft_consensus.cc:2454] T c39106a8bb0146558ee0a3eb67422cae P 60cef69369a34800bf13b929dc656f86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bdafe2bbc0c430f9bdd4e714ef4efa4 in term 1.
I20260927 21:30:33.381896 16354 raft_consensus.cc:2454] T c39106a8bb0146558ee0a3eb67422cae P 46b4f74760a6467ea467ea21c10205b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bdafe2bbc0c430f9bdd4e714ef4efa4 in term 1.
I20260927 21:30:33.382230 16020 leader_election.cc:302] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60cef69369a34800bf13b929dc656f86, 6bdafe2bbc0c430f9bdd4e714ef4efa4; no voters:
I20260927 21:30:33.382586 16424 raft_consensus.cc:2781] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:30:33.383065 16424 raft_consensus.cc:686] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [term 1 LEADER]: Becoming Leader. State: Replica: 6bdafe2bbc0c430f9bdd4e714ef4efa4, State: Running, Role: LEADER
I20260927 21:30:33.383576 16424 consensus_queue.cc:237] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60cef69369a34800bf13b929dc656f86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44853 } } peers { permanent_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37393 } } peers { permanent_uuid: "46b4f74760a6467ea467ea21c10205b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33005 } }
I20260927 21:30:33.388338 15942 catalog_manager.cc:5495] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bdafe2bbc0c430f9bdd4e714ef4efa4 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60cef69369a34800bf13b929dc656f86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46b4f74760a6467ea467ea21c10205b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33005 } health_report { overall_health: UNKNOWN } } }
W20260927 21:30:33.392832 16400 tablet.cc:2369] T c39106a8bb0146558ee0a3eb67422cae P 46b4f74760a6467ea467ea21c10205b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:30:33.412485 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:33.407299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50148 (local address 127.0.97.1:37393)
0927 21:30:33.407522 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:33.407527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:33.407545 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:30:33.407682 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:33.407686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:33.407791 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:33.407917 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:30:33.408252 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.408969 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.410426 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.410823 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.410913 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:33.411366 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:30:33.411898 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:30:33.412006 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:33.412110 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 21:30:33.412180 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 21:30:33.420828 16354 raft_consensus.cc:1261] T c39106a8bb0146558ee0a3eb67422cae P 46b4f74760a6467ea467ea21c10205b4 [term 1 FOLLOWER]: Refusing update from remote peer 6bdafe2bbc0c430f9bdd4e714ef4efa4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:30:33.420845 16221 raft_consensus.cc:1261] T c39106a8bb0146558ee0a3eb67422cae P 60cef69369a34800bf13b929dc656f86 [term 1 FOLLOWER]: Refusing update from remote peer 6bdafe2bbc0c430f9bdd4e714ef4efa4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:30:33.421921 16429 consensus_queue.cc:1035] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60cef69369a34800bf13b929dc656f86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:30:33.422277 16424 consensus_queue.cc:1035] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46b4f74760a6467ea467ea21c10205b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:30:33.434978 16432 mvcc.cc:205] Tried to move back new op lower bound from 7334070818481643520 to 7334070818342973440. Current Snapshot: MvccSnapshot[applied={T|T < 7334070818481643520}]
I20260927 21:30:33.448841 15942 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:30:33.449149 15942 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:30:33.453526 15942 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68: Sending DeleteTablet for 3 replicas of tablet c39106a8bb0146558ee0a3eb67422cae
I20260927 21:30:33.455116 16201 tablet_service.cc:1514] Processing DeleteTablet for tablet c39106a8bb0146558ee0a3eb67422cae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:30:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33138
I20260927 21:30:33.455344 16334 tablet_service.cc:1514] Processing DeleteTablet for tablet c39106a8bb0146558ee0a3eb67422cae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:30:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43106
I20260927 21:30:33.455906 16067 tablet_service.cc:1514] Processing DeleteTablet for tablet c39106a8bb0146558ee0a3eb67422cae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:30:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50144
I20260927 21:30:33.458215 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 16004
I20260927 21:30:33.474047 16443 tablet_replica.cc:330] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4: stopping tablet replica
I20260927 21:30:33.475509 16441 tablet_replica.cc:330] T c39106a8bb0146558ee0a3eb67422cae P 60cef69369a34800bf13b929dc656f86: stopping tablet replica
I20260927 21:30:33.475819 16443 raft_consensus.cc:2229] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:30:33.476682 16443 raft_consensus.cc:2258] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:30:33.477288 16441 raft_consensus.cc:2229] T c39106a8bb0146558ee0a3eb67422cae P 60cef69369a34800bf13b929dc656f86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:30:33.478104 16441 raft_consensus.cc:2258] T c39106a8bb0146558ee0a3eb67422cae P 60cef69369a34800bf13b929dc656f86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:30:33.480564 16443 ts_tablet_manager.cc:1905] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:30:33.481806 16442 tablet_replica.cc:330] T c39106a8bb0146558ee0a3eb67422cae P 46b4f74760a6467ea467ea21c10205b4: stopping tablet replica
I20260927 21:30:33.483546 16441 ts_tablet_manager.cc:1905] T c39106a8bb0146558ee0a3eb67422cae P 60cef69369a34800bf13b929dc656f86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:30:33.484242 16442 raft_consensus.cc:2229] T c39106a8bb0146558ee0a3eb67422cae P 46b4f74760a6467ea467ea21c10205b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:30:33.485594 16442 raft_consensus.cc:2258] T c39106a8bb0146558ee0a3eb67422cae P 46b4f74760a6467ea467ea21c10205b4 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 21:30:33 dist-test-slave-bxm0 krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 21:30:33.488937 16442 ts_tablet_manager.cc:1905] T c39106a8bb0146558ee0a3eb67422cae P 46b4f74760a6467ea467ea21c10205b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:30:33.490252 16443 ts_tablet_manager.cc:1918] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:30:33.490515 16443 log.cc:1196] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/wal/wals/c39106a8bb0146558ee0a3eb67422cae
I20260927 21:30:33.491209 16443 ts_tablet_manager.cc:1939] T c39106a8bb0146558ee0a3eb67422cae P 6bdafe2bbc0c430f9bdd4e714ef4efa4: Deleting consensus metadata
I20260927 21:30:33.493538 15928 catalog_manager.cc:4841] TS 6bdafe2bbc0c430f9bdd4e714ef4efa4 (127.0.97.1:37393): tablet c39106a8bb0146558ee0a3eb67422cae (table test-table [id=dcab75b44e974539bbd77ffcc35ecfbd]) successfully deleted
I20260927 21:30:33.497545 16441 ts_tablet_manager.cc:1918] T c39106a8bb0146558ee0a3eb67422cae P 60cef69369a34800bf13b929dc656f86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:30:33.497771 16441 log.cc:1196] T c39106a8bb0146558ee0a3eb67422cae P 60cef69369a34800bf13b929dc656f86: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/wal/wals/c39106a8bb0146558ee0a3eb67422cae
I20260927 21:30:33.498358 16441 ts_tablet_manager.cc:1939] T c39106a8bb0146558ee0a3eb67422cae P 60cef69369a34800bf13b929dc656f86: Deleting consensus metadata
I20260927 21:30:33.498471 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:33.459480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50154 (local address 127.0.97.1:37393)
0927 21:30:33.459680 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:33.459686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:33.474798 (+ 15112us) server_negotiation.cc:407] Connection header received
0927 21:30:33.475037 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:33.475050 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:33.475171 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:33.475352 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:33.478259 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.479920 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.483077 (+ 3157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.483899 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.484029 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:33.490377 (+ 6348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:33.490396 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:33.490415 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:33.490465 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:33.494694 (+ 4229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:33.495540 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:33.495550 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:33.495553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:33.495729 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:33.496265 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:33.496270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:33.496273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:33.496720 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:33.496996 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:33.497783 (+ 787us) server_negotiation.cc:299] Negotiation successful
0927 21:30:33.498075 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:30:33.500078 16127 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:30:33.500422 15927 catalog_manager.cc:4841] TS 60cef69369a34800bf13b929dc656f86 (127.0.97.2:44853): tablet c39106a8bb0146558ee0a3eb67422cae (table test-table [id=dcab75b44e974539bbd77ffcc35ecfbd]) successfully deleted
I20260927 21:30:33.501603 16442 ts_tablet_manager.cc:1918] T c39106a8bb0146558ee0a3eb67422cae P 46b4f74760a6467ea467ea21c10205b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:30:33.501837 16442 log.cc:1196] T c39106a8bb0146558ee0a3eb67422cae P 46b4f74760a6467ea467ea21c10205b4: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/wal/wals/c39106a8bb0146558ee0a3eb67422cae
I20260927 21:30:33.502418 16442 ts_tablet_manager.cc:1939] T c39106a8bb0146558ee0a3eb67422cae P 46b4f74760a6467ea467ea21c10205b4: Deleting consensus metadata
I20260927 21:30:33.504666 15928 catalog_manager.cc:4841] TS 46b4f74760a6467ea467ea21c10205b4 (127.0.97.3:33005): tablet c39106a8bb0146558ee0a3eb67422cae (table test-table [id=dcab75b44e974539bbd77ffcc35ecfbd]) successfully deleted
I20260927 21:30:33.604007 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 16004
I20260927 21:30:33.629271 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 16138
Sep 27 21:30:33 dist-test-slave-bxm0 krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 21:30:33.646896 16413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:33.630709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33148 (local address 127.0.97.2:44853)
0927 21:30:33.630943 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:33.630950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:33.630975 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:30:33.631124 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:33.631131 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:33.631243 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:33.631354 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:33.632989 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.634029 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.636066 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.636527 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.636646 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:33.641140 (+ 4494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:33.641181 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:33.641198 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:33.641244 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:33.644345 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:33.645057 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:33.645063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:33.645066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:33.645195 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:33.645527 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:33.645530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:33.645532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:33.645980 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:33.646191 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:33.646460 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 21:30:33.646624 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 21:30:33.647549 16261 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:30:33.738090 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 16138
I20260927 21:30:33.757556 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 16271
Sep 27 21:30:33 dist-test-slave-bxm0 krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 21:30:33.772755 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:33.758558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43118 (local address 127.0.97.3:33005)
0927 21:30:33.758760 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:33.758764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:33.758783 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:30:33.758950 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:33.758954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:33.759032 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:33.759135 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:33.760416 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.761144 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.762527 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:33.762926 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:33.763025 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:33.767339 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:33.767358 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:33.767374 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:33.767412 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:33.770179 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:33.770886 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:33.770893 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:33.770895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:33.771037 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:33.771358 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:33.771361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:33.771363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:33.771688 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:33.771919 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:33.772191 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 21:30:33.772382 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 21:30:33.773505 16394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:30:33.866353 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 16271
I20260927 21:30:33.884122 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 15911
I20260927 21:30:33.885301 15973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:30:33.998962 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 15911
I20260927 21:30:54.022274 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46129
--webserver_interface=127.0.97.62
--webserver_port=42619
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:30:54.359771 16449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:30:54.360069 16449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:30:54.360244 16449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:30:54.367802 16449 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 21:30:54.367916 16449 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 21:30:54.368001 16449 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 21:30:54.368077 16449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:30:54.368187 16449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:30:54.368302 16449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:30:54.368403 16449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:30:54.368486 16449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:30:54.378912 16449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:30:54.380286 16449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:30:54.381778 16449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:30:54.404652 16458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:54.404677 16455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:54.404590 16456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:30:54.405947 16449 server_base.cc:1034] running on GCE node
I20260927 21:30:54.406801 16449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:30:54.408396 16449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:30:54.409721 16449 hybrid_clock.cc:648] HybridClock initialized: now 1790544654409665 us; error 59 us; skew 500 ppm
I20260927 21:30:54.410132 16449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:30:54 dist-test-slave-bxm0 krb5kdc[15898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:30:54.418150 16449 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:30:54.419947 16449 webserver.cc:460] Webserver started at http://127.0.97.62:42619/ using document root <none> and password file <none>
I20260927 21:30:54.420802 16449 fs_manager.cc:359] Metadata directory not provided
I20260927 21:30:54.420969 16449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:30:54.427645 16449 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:30:54.430864 16465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:54.432323 16449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:30:54.432538 16449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "218b963fcfda487396ccc45be614fc68"
format_stamp: "Formatted at 2026-09-27 21:30:31 on dist-test-slave-bxm0"
server_key: "51ddf1bd8e2eebf77acb479f6f8d1579"
server_key_iv: "61e116dda024c40246d62d028d1bae0b"
server_key_version: "encryptionkey@0"
I20260927 21:30:54.433684 16449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:30:54.463948 16449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:30:54.469295 16449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:30:54.469645 16449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:30:54.489916 16449 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46129
I20260927 21:30:54.489980 16516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46129 every 8 connection(s)
I20260927 21:30:54.491663 16449 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:30:54.497856 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 16449
I20260927 21:30:54.498951 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:37393
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46129
--webserver_port=41649
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:30:54.500011 16517 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:30:54.505811 16517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68: Bootstrap starting.
I20260927 21:30:54.527015 16517 log.cc:824] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68: Log is configured to *not* fsync() on all Append() calls
I20260927 21:30:54.540169 16517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:30:54.540802 16517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68: Bootstrap complete.
I20260927 21:30:54.550169 16517 raft_consensus.cc:348] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218b963fcfda487396ccc45be614fc68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46129 } }
I20260927 21:30:54.551023 16517 raft_consensus.cc:729] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218b963fcfda487396ccc45be614fc68, State: Initialized, Role: FOLLOWER
I20260927 21:30:54.551789 16517 consensus_queue.cc:260] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218b963fcfda487396ccc45be614fc68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46129 } }
I20260927 21:30:54.552042 16517 raft_consensus.cc:388] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:30:54.552163 16517 raft_consensus.cc:482] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:30:54.552369 16517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:30:54.556429 16517 raft_consensus.cc:504] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218b963fcfda487396ccc45be614fc68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46129 } }
I20260927 21:30:54.556985 16517 leader_election.cc:302] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 218b963fcfda487396ccc45be614fc68; no voters:
I20260927 21:30:54.557719 16517 leader_election.cc:288] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 21:30:54.557871 16521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:30:54.559087 16521 raft_consensus.cc:686] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [term 2 LEADER]: Becoming Leader. State: Replica: 218b963fcfda487396ccc45be614fc68, State: Running, Role: LEADER
I20260927 21:30:54.559720 16521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218b963fcfda487396ccc45be614fc68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46129 } }
I20260927 21:30:54.560094 16517 sys_catalog.cc:564] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:30:54.562399 16522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "218b963fcfda487396ccc45be614fc68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218b963fcfda487396ccc45be614fc68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46129 } } }
I20260927 21:30:54.562898 16522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [sys.catalog]: This master's current role is: LEADER
I20260927 21:30:54.567806 16523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 218b963fcfda487396ccc45be614fc68. Latest consensus state: current_term: 2 leader_uuid: "218b963fcfda487396ccc45be614fc68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218b963fcfda487396ccc45be614fc68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46129 } } }
I20260927 21:30:54.568938 16523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [sys.catalog]: This master's current role is: LEADER
I20260927 21:30:54.570418 16527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:30:54.579347 16527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:30:54.580925 16527 catalog_manager.cc:1259] Loaded cluster ID: 5840b2607efb448692ec1693ad1f32f4
I20260927 21:30:54.581106 16527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:30:54.585255 16527 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:30:54.586994 16527 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68: Loaded TSK: 0
I20260927 21:30:54.608304 16538 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 21:30:54.616711 16527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68: Generated new TSK 1
I20260927 21:30:54.617163 16527 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:30:54.861503 16519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:30:54.861872 16519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:30:54.862056 16519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:30:54.869918 16519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:30:54.870038 16519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:30:54.870215 16519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:30:54.880414 16519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:37393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41649
--tserver_master_addrs=127.0.97.62:46129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:30:54.881829 16519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:30:54.883236 16519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:30:54.898875 16545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:54.898922 16547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:54.899659 16544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:30:54.900106 16519 server_base.cc:1034] running on GCE node
I20260927 21:30:54.900816 16519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:30:54.901901 16519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:30:54.903146 16519 hybrid_clock.cc:648] HybridClock initialized: now 1790544654903094 us; error 45 us; skew 500 ppm
I20260927 21:30:54.903497 16519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:30:54 dist-test-slave-bxm0 krb5kdc[15898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:30:54.907402 16519 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 21:30:54.909075 16519 webserver.cc:460] Webserver started at http://127.0.97.1:41649/ using document root <none> and password file <none>
I20260927 21:30:54.909894 16519 fs_manager.cc:359] Metadata directory not provided
I20260927 21:30:54.910058 16519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:30:54.915973 16519 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 21:30:54.919607 16554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:54.921124 16519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:30:54.921334 16519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4"
format_stamp: "Formatted at 2026-09-27 21:30:31 on dist-test-slave-bxm0"
server_key: "ee68e6ab7ef8d32856e58ac1cb492159"
server_key_iv: "1f464dc9651e30be35e55e8bc50e397f"
server_key_version: "encryptionkey@0"
I20260927 21:30:54.922420 16519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:30:54.949851 16519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:30:54.954560 16519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:30:54.954870 16519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:30:54.955919 16519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:30:54.957310 16519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:30:54.957428 16519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:54.957569 16519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:30:54.957693 16519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 21:30:54.991945 16519 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37393
I20260927 21:30:54.992071 16666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37393 every 8 connection(s)
I20260927 21:30:54.993419 16519 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:30:54.997445 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 16519
I20260927 21:30:54.998309 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44853
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46129
--webserver_port=33923
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:30:55 dist-test-slave-bxm0 krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:30:55.022401 16670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:54.996709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.1:50667)
0927 21:30:54.997951 (+ 1242us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:54.998014 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:55.000884 (+ 2870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:55.002505 (+ 1621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:55.002517 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:55.003372 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:30:55.004207 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:55.004234 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:55.006205 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:55.006209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:55.007545 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:55.007559 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:55.009220 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:55.009226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:55.009427 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:55.010327 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:30:55.010352 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:30:55.012694 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:30:55.017852 (+ 5158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:55.017864 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:55.017883 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:55.018394 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:55.018977 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:55.018981 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:55.018984 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:55.019184 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:55.020249 (+ 1065us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:30:55.020255 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:30:55.020556 (+ 301us) client_negotiation.cc:769] Sending connection context
0927 21:30:55.020810 (+ 254us) client_negotiation.cc:240] Negotiation successful
0927 21:30:55.021227 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"thread_start_us":621,"threads_started":1}
I20260927 21:30:55.022989 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:54.996449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50667 (local address 127.0.97.62:46129)
0927 21:30:54.997265 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:54.997284 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:54.998197 (+ 913us) server_negotiation.cc:407] Connection header received
0927 21:30:55.001481 (+ 3284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:55.001495 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:55.001949 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:55.002278 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:55.004476 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:55.005989 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:55.007737 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:55.008949 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:55.009165 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:55.013043 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:55.013058 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:55.013084 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:55.013164 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:55.017581 (+ 4417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:55.018573 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:55.018585 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:55.018596 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:55.018842 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:55.019307 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:55.019314 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:55.019319 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:55.020077 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:55.020317 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:55.021089 (+ 772us) server_negotiation.cc:299] Negotiation successful
0927 21:30:55.021604 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":377,"threads_started":1}
I20260927 21:30:55.026149 16667 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46129
I20260927 21:30:55.026543 16667 heartbeater.cc:464] Registering TS with master...
I20260927 21:30:55.027455 16667 heartbeater.cc:511] Master 127.0.97.62:46129 requested a full tablet report, sending...
I20260927 21:30:55.029897 16482 ts_manager.cc:194] Registered new tserver with Master: 6bdafe2bbc0c430f9bdd4e714ef4efa4 (127.0.97.1:37393)
I20260927 21:30:55.031401 16482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50667
W20260927 21:30:55.341738 16671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:30:55.342084 16671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:30:55.342255 16671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:30:55.349798 16671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:30:55.349957 16671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:30:55.350142 16671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:30:55.360973 16671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33923
--tserver_master_addrs=127.0.97.62:46129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:30:55.362433 16671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:30:55.363765 16671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:30:55.378611 16676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:55.378762 16679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:55.378767 16677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:30:55.383370 16671 server_base.cc:1034] running on GCE node
I20260927 21:30:55.384109 16671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:30:55.385201 16671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:30:55.386474 16671 hybrid_clock.cc:648] HybridClock initialized: now 1790544655386424 us; error 77 us; skew 500 ppm
I20260927 21:30:55.386842 16671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:30:55 dist-test-slave-bxm0 krb5kdc[15898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:30:55.390079 16671 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 21:30:55.391772 16671 webserver.cc:460] Webserver started at http://127.0.97.2:33923/ using document root <none> and password file <none>
I20260927 21:30:55.392542 16671 fs_manager.cc:359] Metadata directory not provided
I20260927 21:30:55.392715 16671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:30:55.398655 16671 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 21:30:55.401922 16686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:55.403453 16671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 21:30:55.403687 16671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "60cef69369a34800bf13b929dc656f86"
format_stamp: "Formatted at 2026-09-27 21:30:32 on dist-test-slave-bxm0"
server_key: "8b5fa549719d4727a8e15d8ced4b2d45"
server_key_iv: "855ee0abffeceb8073da773360a377b5"
server_key_version: "encryptionkey@0"
I20260927 21:30:55.404743 16671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:30:55.422209 16671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:30:55.427114 16671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:30:55.427433 16671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:30:55.428450 16671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:30:55.429997 16671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:30:55.430121 16671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:55.430264 16671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:30:55.430361 16671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:55.464845 16671 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44853
I20260927 21:30:55.464943 16798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44853 every 8 connection(s)
I20260927 21:30:55.466466 16671 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:30:55.467005 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 16671
I20260927 21:30:55.467864 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:33005
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46129
--webserver_port=33597
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:30:55 dist-test-slave-bxm0 krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:30:55.493465 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:55.469467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38035 (local address 127.0.97.62:46129)
0927 21:30:55.469736 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:55.469745 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:55.471105 (+ 1360us) server_negotiation.cc:407] Connection header received
0927 21:30:55.476462 (+ 5357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:55.476466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:55.476551 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:55.476650 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:55.478654 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:55.479362 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:55.481062 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:55.481493 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:55.481687 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:55.486286 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:55.486291 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:55.486295 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:55.486333 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:55.489245 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:55.490503 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:55.490510 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:55.490516 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:55.490679 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:55.491289 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:55.491294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:55.491297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:55.491702 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:55.491847 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:55.492830 (+ 983us) server_negotiation.cc:299] Negotiation successful
0927 21:30:55.493099 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 21:30:55.494992 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:55.470030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.2:38035)
0927 21:30:55.470902 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:55.470963 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:55.476028 (+ 5065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:55.477023 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:55.477040 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:55.477790 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:30:55.478409 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:55.478429 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:55.479557 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:55.479568 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:55.480880 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:55.480893 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:55.481784 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:55.481792 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:55.482045 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:55.483002 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:30:55.483032 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:30:55.485872 (+ 2840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:30:55.489461 (+ 3589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:55.489475 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:55.489498 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:55.490322 (+ 824us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:55.490824 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:55.490831 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:55.490834 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:55.491135 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:55.491861 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:30:55.491870 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:30:55.492344 (+ 474us) client_negotiation.cc:769] Sending connection context
0927 21:30:55.492691 (+ 347us) client_negotiation.cc:240] Negotiation successful
0927 21:30:55.493372 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":325,"threads_started":1}
I20260927 21:30:55.496867 16799 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46129
I20260927 21:30:55.497216 16799 heartbeater.cc:464] Registering TS with master...
I20260927 21:30:55.498273 16799 heartbeater.cc:511] Master 127.0.97.62:46129 requested a full tablet report, sending...
I20260927 21:30:55.500026 16482 ts_manager.cc:194] Registered new tserver with Master: 60cef69369a34800bf13b929dc656f86 (127.0.97.2:44853)
I20260927 21:30:55.501338 16482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38035
W20260927 21:30:55.824476 16802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:30:55.824777 16802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:30:55.824883 16802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:30:55.832401 16802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:30:55.832523 16802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:30:55.832639 16802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:30:55.842962 16802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:33005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=33597
--tserver_master_addrs=127.0.97.62:46129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:30:55.844235 16802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:30:55.845768 16802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:30:55.860225 16810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:55.860265 16808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:30:55.860383 16807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:30:55.860954 16802 server_base.cc:1034] running on GCE node
I20260927 21:30:55.861696 16802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:30:55.862730 16802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:30:55.863960 16802 hybrid_clock.cc:648] HybridClock initialized: now 1790544655863916 us; error 29 us; skew 500 ppm
I20260927 21:30:55.864255 16802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:30:55 dist-test-slave-bxm0 krb5kdc[15898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:30:55.867364 16802 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 21:30:55.868835 16802 webserver.cc:460] Webserver started at http://127.0.97.3:33597/ using document root <none> and password file <none>
I20260927 21:30:55.869478 16802 fs_manager.cc:359] Metadata directory not provided
I20260927 21:30:55.869596 16802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:30:55.875329 16802 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 21:30:55.877995 16817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:55.879349 16802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 21:30:55.879498 16802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "46b4f74760a6467ea467ea21c10205b4"
format_stamp: "Formatted at 2026-09-27 21:30:33 on dist-test-slave-bxm0"
server_key: "388ac51b07723c9373b0bd6e567650bc"
server_key_iv: "93fe6426bd6bec64db66ca76634e76d4"
server_key_version: "encryptionkey@0"
I20260927 21:30:55.880419 16802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:30:55.911031 16802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:30:55.915534 16802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:30:55.915758 16802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:30:55.916672 16802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:30:55.918030 16802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:30:55.918116 16802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:55.918192 16802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:30:55.918251 16802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:30:55.953110 16802 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33005
I20260927 21:30:55.953218 16929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33005 every 8 connection(s)
I20260927 21:30:55.954715 16802 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:30:55.955252 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 16802
Sep 27 21:30:55 dist-test-slave-bxm0 krb5kdc[15898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:30:55.972818 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:55.956893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58238 (local address 127.0.97.62:46129)
0927 21:30:55.957134 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:55.957141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:55.957976 (+ 835us) server_negotiation.cc:407] Connection header received
0927 21:30:55.958249 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:55.958253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:55.958315 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:55.958399 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:55.960292 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:55.961115 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:55.962853 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:55.963313 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:55.963459 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:55.966457 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:55.966464 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:55.966470 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:55.966537 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:55.969774 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:55.970601 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:55.970607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:55.970610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:55.970755 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:55.971152 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:55.971158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:55.971163 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:55.971503 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:55.971779 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:55.972183 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 21:30:55.972460 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 21:30:55.977794 16933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:55.957529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46149 (local address 127.0.97.62:46129)
0927 21:30:55.958129 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:55.958133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:55.959272 (+ 1139us) server_negotiation.cc:407] Connection header received
0927 21:30:55.961404 (+ 2132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:55.961409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:55.961516 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:55.961698 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:55.963504 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:55.964436 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:55.966268 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:55.967086 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:55.967222 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:55.971309 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:55.971314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:55.971317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:55.971348 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:55.974626 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:55.975584 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:55.975590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:55.975593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:55.975742 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:55.976155 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:55.976158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:55.976160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:55.976440 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:55.976624 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:55.977247 (+ 623us) server_negotiation.cc:299] Negotiation successful
0927 21:30:55.977431 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":390,"threads_started":1}
I20260927 21:30:55.978780 16934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:55.958318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.3:46149)
0927 21:30:55.959122 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:55.959158 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:55.961137 (+ 1979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:55.962246 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:55.962255 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:55.962762 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:30:55.963321 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:55.963338 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:55.964607 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:55.964611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:55.966109 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:55.966126 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:55.967312 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:55.967317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:55.967674 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:55.968350 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:30:55.968373 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:30:55.971031 (+ 2658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:30:55.974765 (+ 3734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:55.974779 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:55.974796 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:55.975428 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:55.975853 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:30:55.975857 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:30:55.975859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:30:55.976044 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:30:55.976597 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:30:55.976602 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:30:55.976924 (+ 322us) client_negotiation.cc:769] Sending connection context
0927 21:30:55.977147 (+ 223us) client_negotiation.cc:240] Negotiation successful
0927 21:30:55.977546 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":328,"threads_started":1}
I20260927 21:30:55.980185 16930 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46129
I20260927 21:30:55.980504 16930 heartbeater.cc:464] Registering TS with master...
I20260927 21:30:55.981230 16930 heartbeater.cc:511] Master 127.0.97.62:46129 requested a full tablet report, sending...
I20260927 21:30:55.982803 16482 ts_manager.cc:194] Registered new tserver with Master: 46b4f74760a6467ea467ea21c10205b4 (127.0.97.3:33005)
I20260927 21:30:55.983726 16482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46149
I20260927 21:30:55.988152 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:30:55.995087 16933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:55.989074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58246 (local address 127.0.97.62:46129)
0927 21:30:55.989302 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:55.989312 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:55.989737 (+ 425us) server_negotiation.cc:407] Connection header received
0927 21:30:55.990005 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:55.990008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:55.990165 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:55.990323 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:30:55.990780 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:55.991542 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:55.993048 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:55.993483 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:55.993621 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:55.994209 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:30:55.994407 (+ 198us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 21:30:55.994798 (+ 391us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58246: token expired
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 21:30:55.995242 16933 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58246: token expired
W20260927 21:30:55.995878 16406 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:46129)
I20260927 21:30:56.008597 16933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:55.996512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58260 (local address 127.0.97.62:46129)
0927 21:30:55.996710 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:55.996715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:55.996731 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:30:55.996772 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:55.996775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:55.996830 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:55.996908 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:55.998308 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:55.999003 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.000337 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.000715 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.000801 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:56.003388 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:56.003393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:56.003396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:56.003436 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:56.006016 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:56.006714 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:56.006718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:56.006720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:56.006871 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:56.007223 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:56.007226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:56.007228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:56.007526 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:56.007729 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:56.007973 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 21:30:56.008141 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 21:30:56.012148 16482 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:30:56.019799 16482 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:30:56.034801 16667 heartbeater.cc:503] Master 127.0.97.62:46129 was elected leader, sending a full tablet report...
I20260927 21:30:56.043388 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:56.032169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33005 (local address 127.0.0.1:39862)
0927 21:30:56.033192 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:56.033245 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:56.033441 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:56.035380 (+ 1939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:56.035386 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:56.035418 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:30:56.035928 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:56.035951 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.038168 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.038176 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:56.040641 (+ 2465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:56.040653 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.042249 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.042255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:56.042546 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:56.042587 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 21:30:56.042745 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 21:30:56.042967 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":49,"thread_start_us":1562,"threads_started":1}
I20260927 21:30:56.044148 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:56.032480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39862 (local address 127.0.97.3:33005)
0927 21:30:56.034628 (+ 2148us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:56.034637 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:56.034670 (+ 33us) server_negotiation.cc:407] Connection header received
0927 21:30:56.034776 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:56.034783 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:56.035151 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:56.035510 (+ 359us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:30:56.036191 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.037951 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.040812 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.042006 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.042158 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:56.042443 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:56.043654 (+ 1211us) server_negotiation.cc:299] Negotiation successful
0927 21:30:56.043850 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1996,"thread_start_us":1833,"threads_started":1}
I20260927 21:30:56.044876 16940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:56.032392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44853 (local address 127.0.0.1:40494)
0927 21:30:56.034015 (+ 1623us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:56.034035 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:56.034142 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:56.035162 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:56.035173 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:56.035220 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:30:56.037249 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:56.037261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.039274 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.039280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:56.041543 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:56.041558 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.044039 (+ 2481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.044046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:56.044220 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:56.044254 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 21:30:56.044348 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 21:30:56.044490 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1466,"mutex_wait_us":4,"thread_start_us":387,"threads_started":1}
I20260927 21:30:56.045570 16944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:56.034127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37393 (local address 127.0.0.1:35336)
0927 21:30:56.036188 (+ 2061us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:56.036931 (+ 743us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:56.037058 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:56.037984 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:56.037989 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:56.038014 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:30:56.038427 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:56.038438 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.040344 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.040349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:56.042737 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:56.042752 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.044855 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.044861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:56.045035 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:56.045066 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 21:30:56.045158 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 21:30:56.045286 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1948,"mutex_wait_us":258,"thread_start_us":1779,"threads_started":1}
I20260927 21:30:56.045758 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:56.033112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40494 (local address 127.0.97.2:44853)
0927 21:30:56.034050 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:56.034063 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:56.034101 (+ 38us) server_negotiation.cc:407] Connection header received
0927 21:30:56.034353 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:56.034361 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:56.034654 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:56.034845 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:30:56.037448 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.039087 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.041783 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.043304 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.043435 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:56.043881 (+ 446us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:56.045215 (+ 1334us) server_negotiation.cc:299] Negotiation successful
0927 21:30:56.045400 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":768,"thread_start_us":661,"threads_started":1}
I20260927 21:30:56.046305 16945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:56.034707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35336 (local address 127.0.97.1:37393)
0927 21:30:56.037098 (+ 2391us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:56.037108 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:56.037134 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:30:56.037217 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:56.037224 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:56.037547 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:56.037761 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:30:56.038643 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.040197 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.042927 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.044367 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.044488 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:56.044702 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:56.045943 (+ 1241us) server_negotiation.cc:299] Negotiation successful
0927 21:30:56.046081 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2220,"thread_start_us":2087,"threads_started":1}
I20260927 21:30:56.048518 16865 tablet_service.cc:1467] Processing CreateTablet for tablet 1f0c45ea35214942b8a9f1671fbe2ee8 (DEFAULT_TABLE table=test-table [id=350c9b604cd94e3cb697048fe72667fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:30:56.049240 16602 tablet_service.cc:1467] Processing CreateTablet for tablet 1f0c45ea35214942b8a9f1671fbe2ee8 (DEFAULT_TABLE table=test-table [id=350c9b604cd94e3cb697048fe72667fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:30:56.050127 16865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0c45ea35214942b8a9f1671fbe2ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:30:56.049654 16734 tablet_service.cc:1467] Processing CreateTablet for tablet 1f0c45ea35214942b8a9f1671fbe2ee8 (DEFAULT_TABLE table=test-table [id=350c9b604cd94e3cb697048fe72667fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:30:56.050547 16602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0c45ea35214942b8a9f1671fbe2ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:30:56.050871 16734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0c45ea35214942b8a9f1671fbe2ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:30:56.063686 16946 tablet_bootstrap.cc:492] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4: Bootstrap starting.
I20260927 21:30:56.067164 16948 tablet_bootstrap.cc:492] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 60cef69369a34800bf13b929dc656f86: Bootstrap starting.
I20260927 21:30:56.068297 16946 tablet_bootstrap.cc:654] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4: Neither blocks nor log segments found. Creating new log.
I20260927 21:30:56.069003 16947 tablet_bootstrap.cc:492] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 46b4f74760a6467ea467ea21c10205b4: Bootstrap starting.
I20260927 21:30:56.069968 16946 log.cc:824] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4: Log is configured to *not* fsync() on all Append() calls
I20260927 21:30:56.071476 16948 tablet_bootstrap.cc:654] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 60cef69369a34800bf13b929dc656f86: Neither blocks nor log segments found. Creating new log.
I20260927 21:30:56.072544 16947 tablet_bootstrap.cc:654] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 46b4f74760a6467ea467ea21c10205b4: Neither blocks nor log segments found. Creating new log.
I20260927 21:30:56.072901 16948 log.cc:824] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 60cef69369a34800bf13b929dc656f86: Log is configured to *not* fsync() on all Append() calls
I20260927 21:30:56.073881 16946 tablet_bootstrap.cc:492] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4: No bootstrap required, opened a new log
I20260927 21:30:56.073876 16947 log.cc:824] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 46b4f74760a6467ea467ea21c10205b4: Log is configured to *not* fsync() on all Append() calls
I20260927 21:30:56.074347 16946 ts_tablet_manager.cc:1397] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260927 21:30:56.076874 16948 tablet_bootstrap.cc:492] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 60cef69369a34800bf13b929dc656f86: No bootstrap required, opened a new log
I20260927 21:30:56.077163 16948 ts_tablet_manager.cc:1397] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 60cef69369a34800bf13b929dc656f86: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260927 21:30:56.077291 16947 tablet_bootstrap.cc:492] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 46b4f74760a6467ea467ea21c10205b4: No bootstrap required, opened a new log
I20260927 21:30:56.077593 16947 ts_tablet_manager.cc:1397] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 46b4f74760a6467ea467ea21c10205b4: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260927 21:30:56.085517 16948 raft_consensus.cc:348] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 60cef69369a34800bf13b929dc656f86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b4f74760a6467ea467ea21c10205b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33005 } } peers { permanent_uuid: "60cef69369a34800bf13b929dc656f86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44853 } } peers { permanent_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37393 } }
I20260927 21:30:56.086071 16948 raft_consensus.cc:374] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 60cef69369a34800bf13b929dc656f86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:30:56.086252 16948 raft_consensus.cc:729] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 60cef69369a34800bf13b929dc656f86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60cef69369a34800bf13b929dc656f86, State: Initialized, Role: FOLLOWER
I20260927 21:30:56.086374 16946 raft_consensus.cc:348] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b4f74760a6467ea467ea21c10205b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33005 } } peers { permanent_uuid: "60cef69369a34800bf13b929dc656f86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44853 } } peers { permanent_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37393 } }
I20260927 21:30:56.086892 16946 raft_consensus.cc:374] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:30:56.087036 16946 raft_consensus.cc:729] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bdafe2bbc0c430f9bdd4e714ef4efa4, State: Initialized, Role: FOLLOWER
I20260927 21:30:56.087013 16948 consensus_queue.cc:260] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 60cef69369a34800bf13b929dc656f86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b4f74760a6467ea467ea21c10205b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33005 } } peers { permanent_uuid: "60cef69369a34800bf13b929dc656f86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44853 } } peers { permanent_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37393 } }
I20260927 21:30:56.087807 16946 consensus_queue.cc:260] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b4f74760a6467ea467ea21c10205b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33005 } } peers { permanent_uuid: "60cef69369a34800bf13b929dc656f86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44853 } } peers { permanent_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37393 } }
I20260927 21:30:56.088341 16799 heartbeater.cc:503] Master 127.0.97.62:46129 was elected leader, sending a full tablet report...
I20260927 21:30:56.089133 16948 ts_tablet_manager.cc:1428] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 60cef69369a34800bf13b929dc656f86: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
I20260927 21:30:56.089872 16947 raft_consensus.cc:348] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 46b4f74760a6467ea467ea21c10205b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b4f74760a6467ea467ea21c10205b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33005 } } peers { permanent_uuid: "60cef69369a34800bf13b929dc656f86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44853 } } peers { permanent_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37393 } }
I20260927 21:30:56.090442 16947 raft_consensus.cc:374] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 46b4f74760a6467ea467ea21c10205b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:30:56.090606 16947 raft_consensus.cc:729] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 46b4f74760a6467ea467ea21c10205b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46b4f74760a6467ea467ea21c10205b4, State: Initialized, Role: FOLLOWER
I20260927 21:30:56.091023 16946 ts_tablet_manager.cc:1428] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260927 21:30:56.091411 16947 consensus_queue.cc:260] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 46b4f74760a6467ea467ea21c10205b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b4f74760a6467ea467ea21c10205b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33005 } } peers { permanent_uuid: "60cef69369a34800bf13b929dc656f86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44853 } } peers { permanent_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37393 } }
I20260927 21:30:56.092679 16930 heartbeater.cc:503] Master 127.0.97.62:46129 was elected leader, sending a full tablet report...
I20260927 21:30:56.093689 16947 ts_tablet_manager.cc:1428] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 46b4f74760a6467ea467ea21c10205b4: Time spent starting tablet: real 0.016s user 0.010s sys 0.006s
I20260927 21:30:56.134030 16953 raft_consensus.cc:482] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:30:56.134397 16953 raft_consensus.cc:504] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b4f74760a6467ea467ea21c10205b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33005 } } peers { permanent_uuid: "60cef69369a34800bf13b929dc656f86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44853 } } peers { permanent_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37393 } }
I20260927 21:30:56.136159 16953 leader_election.cc:288] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46b4f74760a6467ea467ea21c10205b4 (127.0.97.3:33005), 60cef69369a34800bf13b929dc656f86 (127.0.97.2:44853)
I20260927 21:30:56.143299 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:56.136752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38741 (local address 127.0.97.2:44853)
0927 21:30:56.137090 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:56.137099 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:56.137580 (+ 481us) server_negotiation.cc:407] Connection header received
0927 21:30:56.137867 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:56.137873 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:56.137963 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:56.138071 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:30:56.138703 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.139507 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.141358 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.142310 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.142477 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:56.142578 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:56.142877 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 21:30:56.143006 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260927 21:30:56.143374 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:56.136659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38957 (local address 127.0.97.3:33005)
0927 21:30:56.136873 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:56.136880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:56.137852 (+ 972us) server_negotiation.cc:407] Connection header received
0927 21:30:56.138135 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:56.138140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:56.138197 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:56.138303 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:30:56.138963 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.139700 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.141501 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.142432 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.142584 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:56.142698 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:56.142979 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 21:30:56.143105 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 21:30:56.143543 16955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:56.136518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44853 (local address 127.0.97.1:38741)
0927 21:30:56.137451 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:56.137470 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:56.137704 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:56.138088 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:56.138093 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:56.138132 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:30:56.138554 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:56.138564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.139647 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.139654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:56.141180 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:56.141192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.142529 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.142535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:56.142709 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:56.142733 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 21:30:56.142850 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 21:30:56.142985 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":1,"thread_start_us":535,"threads_started":1}
I20260927 21:30:56.143517 16956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:56.136801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33005 (local address 127.0.97.1:38957)
0927 21:30:56.137725 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0927 21:30:56.137739 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:30:56.137982 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:30:56.138320 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:30:56.138325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:30:56.138353 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:30:56.138769 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:56.138777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.139851 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.139856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:56.141319 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:30:56.141330 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.142659 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.142665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:30:56.142815 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:56.142837 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 21:30:56.142954 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 21:30:56.143083 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":618,"threads_started":1}
I20260927 21:30:56.144706 16754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0c45ea35214942b8a9f1671fbe2ee8" candidate_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60cef69369a34800bf13b929dc656f86" is_pre_election: true
I20260927 21:30:56.144985 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0c45ea35214942b8a9f1671fbe2ee8" candidate_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46b4f74760a6467ea467ea21c10205b4" is_pre_election: true
I20260927 21:30:56.145352 16754 raft_consensus.cc:2454] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 60cef69369a34800bf13b929dc656f86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bdafe2bbc0c430f9bdd4e714ef4efa4 in term 0.
I20260927 21:30:56.145504 16885 raft_consensus.cc:2454] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 46b4f74760a6467ea467ea21c10205b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bdafe2bbc0c430f9bdd4e714ef4efa4 in term 0.
I20260927 21:30:56.146112 16555 leader_election.cc:302] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60cef69369a34800bf13b929dc656f86, 6bdafe2bbc0c430f9bdd4e714ef4efa4; no voters:
I20260927 21:30:56.146591 16953 raft_consensus.cc:2781] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:30:56.146746 16953 raft_consensus.cc:482] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:30:56.146875 16953 raft_consensus.cc:3037] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:30:56.149138 16953 raft_consensus.cc:504] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b4f74760a6467ea467ea21c10205b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33005 } } peers { permanent_uuid: "60cef69369a34800bf13b929dc656f86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44853 } } peers { permanent_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37393 } }
I20260927 21:30:56.149856 16953 leader_election.cc:288] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [CANDIDATE]: Term 1 election: Requested vote from peers 46b4f74760a6467ea467ea21c10205b4 (127.0.97.3:33005), 60cef69369a34800bf13b929dc656f86 (127.0.97.2:44853)
I20260927 21:30:56.150449 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0c45ea35214942b8a9f1671fbe2ee8" candidate_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46b4f74760a6467ea467ea21c10205b4"
I20260927 21:30:56.150449 16754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0c45ea35214942b8a9f1671fbe2ee8" candidate_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60cef69369a34800bf13b929dc656f86"
I20260927 21:30:56.150671 16754 raft_consensus.cc:3037] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 60cef69369a34800bf13b929dc656f86 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:30:56.150671 16885 raft_consensus.cc:3037] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 46b4f74760a6467ea467ea21c10205b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:30:56.154016 16885 raft_consensus.cc:2454] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 46b4f74760a6467ea467ea21c10205b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bdafe2bbc0c430f9bdd4e714ef4efa4 in term 1.
I20260927 21:30:56.154016 16754 raft_consensus.cc:2454] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 60cef69369a34800bf13b929dc656f86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bdafe2bbc0c430f9bdd4e714ef4efa4 in term 1.
I20260927 21:30:56.154560 16555 leader_election.cc:302] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 46b4f74760a6467ea467ea21c10205b4, 60cef69369a34800bf13b929dc656f86, 6bdafe2bbc0c430f9bdd4e714ef4efa4; no voters:
I20260927 21:30:56.154987 16953 raft_consensus.cc:2781] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:30:56.155478 16953 raft_consensus.cc:686] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [term 1 LEADER]: Becoming Leader. State: Replica: 6bdafe2bbc0c430f9bdd4e714ef4efa4, State: Running, Role: LEADER
I20260927 21:30:56.156013 16953 consensus_queue.cc:237] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b4f74760a6467ea467ea21c10205b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33005 } } peers { permanent_uuid: "60cef69369a34800bf13b929dc656f86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44853 } } peers { permanent_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37393 } }
I20260927 21:30:56.160604 16481 catalog_manager.cc:5495] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bdafe2bbc0c430f9bdd4e714ef4efa4 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b4f74760a6467ea467ea21c10205b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60cef69369a34800bf13b929dc656f86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37393 } health_report { overall_health: HEALTHY } } }
I20260927 21:30:56.181708 16945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:56.176149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35348 (local address 127.0.97.1:37393)
0927 21:30:56.176374 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:56.176379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:56.176401 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:30:56.176648 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:56.176652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:56.176730 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:56.176896 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:30:56.177326 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.178117 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.179495 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.179911 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.180061 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:56.180568 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:30:56.181059 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:30:56.181227 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:56.181298 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 21:30:56.181384 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 21:30:56.190932 16754 raft_consensus.cc:1261] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 60cef69369a34800bf13b929dc656f86 [term 1 FOLLOWER]: Refusing update from remote peer 6bdafe2bbc0c430f9bdd4e714ef4efa4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:30:56.190932 16885 raft_consensus.cc:1261] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 46b4f74760a6467ea467ea21c10205b4 [term 1 FOLLOWER]: Refusing update from remote peer 6bdafe2bbc0c430f9bdd4e714ef4efa4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:30:56.192066 16957 consensus_queue.cc:1035] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46b4f74760a6467ea467ea21c10205b4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:30:56.192488 16953 consensus_queue.cc:1035] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60cef69369a34800bf13b929dc656f86" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:30:56.204977 16959 mvcc.cc:205] Tried to move back new op lower bound from 7334070911746232320 to 7334070911618826240. Current Snapshot: MvccSnapshot[applied={T|T < 7334070911746232320}]
I20260927 21:30:56.205730 16960 mvcc.cc:205] Tried to move back new op lower bound from 7334070911746232320 to 7334070911618826240. Current Snapshot: MvccSnapshot[applied={T|T < 7334070911746232320}]
I20260927 21:30:56.220077 16480 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:30:56.220418 16480 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:30:56.224383 16480 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68: Sending DeleteTablet for 3 replicas of tablet 1f0c45ea35214942b8a9f1671fbe2ee8
I20260927 21:30:56.225843 16865 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f0c45ea35214942b8a9f1671fbe2ee8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:30:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39862
I20260927 21:30:56.225835 16734 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f0c45ea35214942b8a9f1671fbe2ee8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:30:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40494
I20260927 21:30:56.226176 16602 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f0c45ea35214942b8a9f1671fbe2ee8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:30:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35336
I20260927 21:30:56.226990 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 16519
I20260927 21:30:56.238519 16970 tablet_replica.cc:330] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 60cef69369a34800bf13b929dc656f86: stopping tablet replica
I20260927 21:30:56.238737 16971 tablet_replica.cc:330] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4: stopping tablet replica
I20260927 21:30:56.239329 16971 raft_consensus.cc:2229] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:30:56.239312 16970 raft_consensus.cc:2229] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 60cef69369a34800bf13b929dc656f86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:30:56.240000 16970 raft_consensus.cc:2258] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 60cef69369a34800bf13b929dc656f86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:30:56.240029 16971 raft_consensus.cc:2258] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:30:56.242591 16970 ts_tablet_manager.cc:1905] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 60cef69369a34800bf13b929dc656f86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:30:56.242597 16971 ts_tablet_manager.cc:1905] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:30:56.250639 16971 ts_tablet_manager.cc:1918] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:30:56.250715 16970 ts_tablet_manager.cc:1918] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 60cef69369a34800bf13b929dc656f86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:30:56.250919 16971 log.cc:1196] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/wal/wals/1f0c45ea35214942b8a9f1671fbe2ee8
I20260927 21:30:56.251044 16970 log.cc:1196] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 60cef69369a34800bf13b929dc656f86: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/wal/wals/1f0c45ea35214942b8a9f1671fbe2ee8
I20260927 21:30:56.251575 16971 ts_tablet_manager.cc:1939] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 6bdafe2bbc0c430f9bdd4e714ef4efa4: Deleting consensus metadata
I20260927 21:30:56.251703 16970 ts_tablet_manager.cc:1939] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 60cef69369a34800bf13b929dc656f86: Deleting consensus metadata
I20260927 21:30:56.253672 16467 catalog_manager.cc:4841] TS 6bdafe2bbc0c430f9bdd4e714ef4efa4 (127.0.97.1:37393): tablet 1f0c45ea35214942b8a9f1671fbe2ee8 (table test-table [id=350c9b604cd94e3cb697048fe72667fc]) successfully deleted
I20260927 21:30:56.253865 16466 catalog_manager.cc:4841] TS 60cef69369a34800bf13b929dc656f86 (127.0.97.2:44853): tablet 1f0c45ea35214942b8a9f1671fbe2ee8 (table test-table [id=350c9b604cd94e3cb697048fe72667fc]) successfully deleted
I20260927 21:30:56.254380 16969 tablet_replica.cc:330] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 46b4f74760a6467ea467ea21c10205b4: stopping tablet replica
I20260927 21:30:56.254884 16969 raft_consensus.cc:2229] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 46b4f74760a6467ea467ea21c10205b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:30:56.255450 16969 raft_consensus.cc:2258] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 46b4f74760a6467ea467ea21c10205b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:30:56.255822 16945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:56.228679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35360 (local address 127.0.97.1:37393)
0927 21:30:56.228901 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:56.228908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:56.228930 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:30:56.229002 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:56.229009 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:56.229114 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:56.229243 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:56.231697 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.233294 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.236066 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.237055 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.237214 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:56.245760 (+ 8546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:56.245777 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:56.245796 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:56.245854 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:56.250373 (+ 4519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:56.251562 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:56.251573 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:56.251576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:56.251790 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:56.253234 (+ 1444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:56.253240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:56.253243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:56.254287 (+ 1044us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:56.254624 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:56.255008 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 21:30:56.255267 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 21:30:56.256433 16662 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:30:56.257717 16969 ts_tablet_manager.cc:1905] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 46b4f74760a6467ea467ea21c10205b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:30:56.264199 16969 ts_tablet_manager.cc:1918] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 46b4f74760a6467ea467ea21c10205b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:30:56.264346 16969 log.cc:1196] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 46b4f74760a6467ea467ea21c10205b4: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/wal/wals/1f0c45ea35214942b8a9f1671fbe2ee8
I20260927 21:30:56.264824 16969 ts_tablet_manager.cc:1939] T 1f0c45ea35214942b8a9f1671fbe2ee8 P 46b4f74760a6467ea467ea21c10205b4: Deleting consensus metadata
I20260927 21:30:56.266486 16467 catalog_manager.cc:4841] TS 46b4f74760a6467ea467ea21c10205b4 (127.0.97.3:33005): tablet 1f0c45ea35214942b8a9f1671fbe2ee8 (table test-table [id=350c9b604cd94e3cb697048fe72667fc]) successfully deleted
I20260927 21:30:56.358994 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 16519
I20260927 21:30:56.377787 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 16671
I20260927 21:30:56.396332 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:56.378537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40508 (local address 127.0.97.2:44853)
0927 21:30:56.378723 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:56.378731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:56.378768 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:30:56.378985 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:56.378991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:56.379071 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:56.379185 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:56.380544 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.381311 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.382989 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.383398 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.383540 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:56.390493 (+ 6953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:56.390510 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:56.390525 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:56.390566 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:56.393260 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:56.394158 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:56.394168 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:56.394172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:56.394317 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:56.394711 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:56.394716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:56.394719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:56.395115 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:56.395430 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:56.395750 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 21:30:56.395969 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 21:30:56.397110 16794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:30:56.489564 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 16671
I20260927 21:30:56.506958 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 16802
I20260927 21:30:56.520282 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:30:56.507808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39878 (local address 127.0.97.3:33005)
0927 21:30:56.507980 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 21:30:56.507985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:30:56.508003 (+ 18us) server_negotiation.cc:407] Connection header received
0927 21:30:56.508159 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:30:56.508163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:30:56.508227 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:30:56.508330 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:30:56.509784 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.510511 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.511752 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:30:56.512136 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:30:56.512226 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:30:56.514972 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:30:56.514986 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:30:56.515003 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:30:56.515037 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:30:56.517786 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:56.518476 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:56.518483 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:56.518486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:56.518621 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:30:56.518946 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:30:56.518949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:30:56.518951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:30:56.519268 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:30:56.519477 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 21:30:56.519751 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 21:30:56.519922 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 21:30:56.520959 16925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:30:56.611603 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 16802
I20260927 21:30:56.628397 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 16449
I20260927 21:30:56.629685 16512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:30:56.743389 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 16449
I20260927 21:31:16.765342 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46129
--webserver_interface=127.0.97.62
--webserver_port=42619
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:31:17.105954 16976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:17.106323 16976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:17.106487 16976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:17.113859 16976 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 21:31:17.114001 16976 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 21:31:17.114092 16976 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 21:31:17.114190 16976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:31:17.114287 16976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:17.114406 16976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:17.114513 16976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:31:17.114596 16976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:31:17.125803 16976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:17.127240 16976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:17.128605 16976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:17.141122 16984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:17.141193 16982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:17.141175 16981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:17.141484 16976 server_base.cc:1034] running on GCE node
I20260927 21:31:17.142619 16976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:17.144145 16976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:17.145504 16976 hybrid_clock.cc:648] HybridClock initialized: now 1790544677145439 us; error 44 us; skew 500 ppm
I20260927 21:31:17.145890 16976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:31:17 dist-test-slave-bxm0 krb5kdc[15898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:31:17.153105 16976 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:31:17.154793 16976 webserver.cc:460] Webserver started at http://127.0.97.62:42619/ using document root <none> and password file <none>
I20260927 21:31:17.155544 16976 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:17.155676 16976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:17.161309 16976 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 21:31:17.164412 16991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:17.166024 16976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:31:17.166215 16976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "218b963fcfda487396ccc45be614fc68"
format_stamp: "Formatted at 2026-09-27 21:30:31 on dist-test-slave-bxm0"
server_key: "51ddf1bd8e2eebf77acb479f6f8d1579"
server_key_iv: "61e116dda024c40246d62d028d1bae0b"
server_key_version: "encryptionkey@0"
I20260927 21:31:17.167207 16976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:17.186066 16976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:17.190277 16976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:17.190580 16976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:17.209414 16976 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46129
I20260927 21:31:17.209475 17042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46129 every 8 connection(s)
I20260927 21:31:17.211059 16976 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:31:17.211593 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 16976
I20260927 21:31:17.212663 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:37393
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46129
--webserver_port=41649
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:31:17.218835 17043 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:31:17.222406 17043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68: Bootstrap starting.
I20260927 21:31:17.243046 17043 log.cc:824] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68: Log is configured to *not* fsync() on all Append() calls
I20260927 21:31:17.258814 17043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:31:17.259392 17043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68: Bootstrap complete.
I20260927 21:31:17.268702 17043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218b963fcfda487396ccc45be614fc68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46129 } }
I20260927 21:31:17.269693 17043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218b963fcfda487396ccc45be614fc68, State: Initialized, Role: FOLLOWER
I20260927 21:31:17.270403 17043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218b963fcfda487396ccc45be614fc68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46129 } }
I20260927 21:31:17.270735 17043 raft_consensus.cc:388] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:31:17.270859 17043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:31:17.271059 17043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [term 2 FOLLOWER]: Advancing to term 3
I20260927 21:31:17.275405 17043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218b963fcfda487396ccc45be614fc68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46129 } }
I20260927 21:31:17.275970 17043 leader_election.cc:302] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 218b963fcfda487396ccc45be614fc68; no voters:
I20260927 21:31:17.276659 17043 leader_election.cc:288] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 21:31:17.276809 17047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 21:31:17.277961 17047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [term 3 LEADER]: Becoming Leader. State: Replica: 218b963fcfda487396ccc45be614fc68, State: Running, Role: LEADER
I20260927 21:31:17.278530 17047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218b963fcfda487396ccc45be614fc68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46129 } }
I20260927 21:31:17.279016 17043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:31:17.281435 17049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 218b963fcfda487396ccc45be614fc68. Latest consensus state: current_term: 3 leader_uuid: "218b963fcfda487396ccc45be614fc68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218b963fcfda487396ccc45be614fc68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46129 } } }
I20260927 21:31:17.282035 17049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [sys.catalog]: This master's current role is: LEADER
I20260927 21:31:17.281407 17048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "218b963fcfda487396ccc45be614fc68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218b963fcfda487396ccc45be614fc68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46129 } } }
I20260927 21:31:17.282618 17048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68 [sys.catalog]: This master's current role is: LEADER
I20260927 21:31:17.292062 17052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:31:17.304703 17052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:31:17.306236 17052 catalog_manager.cc:1259] Loaded cluster ID: 5840b2607efb448692ec1693ad1f32f4
I20260927 21:31:17.306360 17052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:31:17.311252 17052 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:31:17.312577 17052 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68: Loaded TSK: 0
I20260927 21:31:17.312731 17052 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 218b963fcfda487396ccc45be614fc68: Loaded TSK: 1
F20260927 21:31:17.313555 17052 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6f5d50f9ad google::LogMessage::Fail() at ??:0
@ 0x7f6f5d51288a google::LogMessage::SendToLog() at ??:0
@ 0x7f6f5d50f4de google::LogMessage::Flush() at ??:0
@ 0x7f6f5d510a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6f612737d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f6f6126273e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f6f72f3e1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f6f72f3d783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f6f72fc92a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f6f72fc90c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f6f730135e9 std::function<>::operator()() at ??:0
@ 0x7f6f72f41cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f6f72f3fc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6f72fc7610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f6f72fc7439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6f6015b345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6f6015e91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f6f6015e799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6f60139eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6f634136db start_thread at ??:0
@ 0x7f6f5c3a961f clone at ??:0
W20260927 21:31:17.613111 17045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:17.613445 17045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:17.613615 17045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:17.621348 17045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:17.621474 17045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:17.621706 17045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:31:17.635036 17045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:37393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41649
--tserver_master_addrs=127.0.97.62:46129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:17.636411 17045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:17.637840 17045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:17.651356 17090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:17.651290 17091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:17.651331 17093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:17.652192 17045 server_base.cc:1034] running on GCE node
I20260927 21:31:17.652818 17045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:17.653878 17045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:17.655102 17045 hybrid_clock.cc:648] HybridClock initialized: now 1790544677655064 us; error 49 us; skew 500 ppm
I20260927 21:31:17.655431 17045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:31:17 dist-test-slave-bxm0 krb5kdc[15898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:31:17.658560 17045 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 21:31:17.660053 17045 webserver.cc:460] Webserver started at http://127.0.97.1:41649/ using document root <none> and password file <none>
I20260927 21:31:17.660753 17045 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:17.660907 17045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:17.667001 17045 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 21:31:17.669898 17100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:17.671295 17045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:31:17.671473 17045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "6bdafe2bbc0c430f9bdd4e714ef4efa4"
format_stamp: "Formatted at 2026-09-27 21:30:31 on dist-test-slave-bxm0"
server_key: "ee68e6ab7ef8d32856e58ac1cb492159"
server_key_iv: "1f464dc9651e30be35e55e8bc50e397f"
server_key_version: "encryptionkey@0"
I20260927 21:31:17.672502 17045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:17.693709 17045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:17.698297 17045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:17.698601 17045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:17.699671 17045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:31:17.701227 17045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:31:17.701350 17045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:17.701485 17045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:31:17.701570 17045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:17.736111 17045 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37393
I20260927 21:31:17.736159 17212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37393 every 8 connection(s)
I20260927 21:31:17.737600 17045 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:31:17.742458 17215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:17.740598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.1:54885)
0927 21:31:17.741322 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:17.741514 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":282,"threads_started":1}
W20260927 21:31:17.743165 17213 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46129 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46129: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
I20260927 21:31:17.744058 17215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:17.743709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.1:37169)
0927 21:31:17.743886 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:17.743954 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 21:31:17.744918 17215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:17.744599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.1:54617)
0927 21:31:17.744735 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:17.744810 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 21:31:17.745193 17213 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:31:17.747965 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 17045
I20260927 21:31:17.748710 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44853
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46129
--webserver_port=33923
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:31:18.097081 17216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:18.097368 17216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:18.097523 17216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:18.105044 17216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:18.105199 17216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:18.105370 17216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:31:18.115320 17216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33923
--tserver_master_addrs=127.0.97.62:46129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:18.116631 17216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:18.118008 17216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:18.133400 17221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:18.133400 17222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:18.134452 17224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:18.134759 17216 server_base.cc:1034] running on GCE node
I20260927 21:31:18.135452 17216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:18.136611 17216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:18.137847 17216 hybrid_clock.cc:648] HybridClock initialized: now 1790544678137776 us; error 59 us; skew 500 ppm
I20260927 21:31:18.138206 17216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:31:18 dist-test-slave-bxm0 krb5kdc[15898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:31:18.141831 17216 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 21:31:18.143441 17216 webserver.cc:460] Webserver started at http://127.0.97.2:33923/ using document root <none> and password file <none>
I20260927 21:31:18.144198 17216 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:18.144354 17216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:18.150277 17216 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 21:31:18.153136 17231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:18.154714 17216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:31:18.154901 17216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "60cef69369a34800bf13b929dc656f86"
format_stamp: "Formatted at 2026-09-27 21:30:32 on dist-test-slave-bxm0"
server_key: "8b5fa549719d4727a8e15d8ced4b2d45"
server_key_iv: "855ee0abffeceb8073da773360a377b5"
server_key_version: "encryptionkey@0"
I20260927 21:31:18.155927 17216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:18.176426 17216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:18.180723 17216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:18.181002 17216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:18.182103 17216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:31:18.183452 17216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:31:18.183581 17216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:18.183718 17216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:31:18.183801 17216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:18.216645 17216 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44853
I20260927 21:31:18.216696 17343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44853 every 8 connection(s)
I20260927 21:31:18.218137 17216 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:31:18.223035 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:18.221099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.2:37571)
0927 21:31:18.221942 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:18.222153 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":288,"threads_started":1}
W20260927 21:31:18.223755 17344 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46129 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46129: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
I20260927 21:31:18.224737 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:18.224334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.2:37103)
0927 21:31:18.224518 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:18.224620 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 21:31:18.225940 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:18.225279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.2:57747)
0927 21:31:18.225429 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:18.225515 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 21:31:18.226262 17344 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:31:18.226606 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 17216
I20260927 21:31:18.227254 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:33005
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46129
--webserver_port=33597
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:31:18.550917 17347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:18.551198 17347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:18.551363 17347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:18.558507 17347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:18.558616 17347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:18.558826 17347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:31:18.568751 17347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:33005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=33597
--tserver_master_addrs=127.0.97.62:46129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:18.570106 17347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:18.571390 17347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:18.584857 17352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:18.584900 17355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:18.584813 17353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:18.585965 17347 server_base.cc:1034] running on GCE node
I20260927 21:31:18.586604 17347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:18.587605 17347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:18.588833 17347 hybrid_clock.cc:648] HybridClock initialized: now 1790544678588785 us; error 61 us; skew 500 ppm
I20260927 21:31:18.589201 17347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:31:18 dist-test-slave-bxm0 krb5kdc[15898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:31:18.592268 17347 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 21:31:18.593825 17347 webserver.cc:460] Webserver started at http://127.0.97.3:33597/ using document root <none> and password file <none>
I20260927 21:31:18.594542 17347 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:18.594689 17347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:18.601056 17347 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 21:31:18.605008 17362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:18.606774 17347 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260927 21:31:18.606961 17347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "46b4f74760a6467ea467ea21c10205b4"
format_stamp: "Formatted at 2026-09-27 21:30:33 on dist-test-slave-bxm0"
server_key: "388ac51b07723c9373b0bd6e567650bc"
server_key_iv: "93fe6426bd6bec64db66ca76634e76d4"
server_key_version: "encryptionkey@0"
I20260927 21:31:18.608033 17347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:18.619992 17347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:18.624046 17347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:18.624320 17347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:18.625277 17347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:31:18.626603 17347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:31:18.626713 17347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:18.626843 17347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:31:18.626924 17347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:18.659431 17347 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33005
I20260927 21:31:18.659477 17474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33005 every 8 connection(s)
I20260927 21:31:18.661057 17347 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:31:18.665938 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:18.664027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.3:56537)
0927 21:31:18.664766 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:18.665004 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":285,"threads_started":1}
W20260927 21:31:18.666538 17475 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46129 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46129: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
I20260927 21:31:18.667321 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:18.667018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.3:50347)
0927 21:31:18.667142 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:18.667207 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 21:31:18.668103 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:18.667841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.3:47025)
0927 21:31:18.667944 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:18.668009 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 21:31:18.668371 17475 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:31:18.670650 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 17347
I20260927 21:31:18.746462 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:18.745743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.1:54277)
0927 21:31:18.746183 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:18.746266 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":200,"threads_started":1}
I20260927 21:31:19.227636 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:19.226847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.2:42413)
0927 21:31:19.227306 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:19.227396 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":222,"threads_started":1}
I20260927 21:31:19.669961 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:19.668941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.3:49711)
0927 21:31:19.669460 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:19.669605 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":242,"threads_started":1}
I20260927 21:31:19.748363 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:19.747498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.1:59891)
0927 21:31:19.748031 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:19.748115 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":247,"threads_started":1}
I20260927 21:31:20.229506 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:20.228670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.2:48821)
0927 21:31:20.229235 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:20.229332 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":311,"threads_started":1}
I20260927 21:31:20.671531 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:20.670825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.3:40711)
0927 21:31:20.671286 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:20.671361 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":246,"threads_started":1}
I20260927 21:31:20.749970 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:20.749245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.1:37471)
0927 21:31:20.749710 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:20.749795 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":218,"threads_started":1}
I20260927 21:31:21.231412 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:21.230553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.2:42181)
0927 21:31:21.231095 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:21.231200 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":293,"threads_started":1}
I20260927 21:31:21.673400 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:21.672550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.3:50023)
0927 21:31:21.673121 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:21.673225 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":334,"threads_started":1}
I20260927 21:31:21.751784 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:21.751006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.1:43221)
0927 21:31:21.751511 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:21.751601 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":255,"threads_started":1}
I20260927 21:31:22.233213 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:22.232461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.2:44081)
0927 21:31:22.232929 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:22.233015 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":234,"threads_started":1}
I20260927 21:31:22.675374 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:22.674518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.3:57737)
0927 21:31:22.675032 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:22.675159 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":244,"threads_started":1}
I20260927 21:31:22.753808 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:22.752839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.1:52009)
0927 21:31:22.753431 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:22.753535 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":319,"threads_started":1}
I20260927 21:31:23.235185 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:23.234360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.2:34365)
0927 21:31:23.234905 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:23.234996 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":274,"threads_started":1}
I20260927 21:31:23.677040 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:23.676318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.3:51197)
0927 21:31:23.676792 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:23.676866 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":259,"threads_started":1}
I20260927 21:31:23.755415 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:23.754739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.1:50383)
0927 21:31:23.755171 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:23.755252 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":227,"threads_started":1}
I20260927 21:31:24.236835 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:24.236107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.2:45011)
0927 21:31:24.236577 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:24.236669 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":226,"threads_started":1}
I20260927 21:31:24.678653 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:24.677941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.3:39083)
0927 21:31:24.678391 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:24.678479 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":240,"threads_started":1}
I20260927 21:31:24.757233 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:24.756324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.1:59961)
0927 21:31:24.756853 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:24.756983 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":200,"threads_started":1}
I20260927 21:31:25.238641 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:25.237860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.2:33209)
0927 21:31:25.238349 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:25.238427 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":213,"threads_started":1}
I20260927 21:31:25.680296 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:25.679529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.3:44175)
0927 21:31:25.679998 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:25.680078 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":213,"threads_started":1}
I20260927 21:31:25.759001 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:25.758301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.1:57671)
0927 21:31:25.758740 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:25.758819 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":239,"threads_started":1}
I20260927 21:31:26.240180 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:26.239517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.2:36801)
0927 21:31:26.239939 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:26.240019 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":216,"threads_started":1}
I20260927 21:31:26.681991 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:26.681203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.3:58237)
0927 21:31:26.681736 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:26.681830 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":248,"threads_started":1}
I20260927 21:31:26.760807 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:26.760031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.1:58111)
0927 21:31:26.760478 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:26.760556 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":230,"threads_started":1}
I20260927 21:31:27.242161 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:27.241323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.2:39105)
0927 21:31:27.241905 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:27.241982 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":286,"threads_started":1}
I20260927 21:31:27.683591 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:27.682907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.3:46887)
0927 21:31:27.683348 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:27.683423 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":223,"threads_started":1}
I20260927 21:31:27.762652 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:27.761885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.1:43293)
0927 21:31:27.762365 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:27.762444 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":245,"threads_started":1}
I20260927 21:31:28.244015 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:28.243137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.2:42095)
0927 21:31:28.243706 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:28.243795 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":341,"threads_started":1}
I20260927 21:31:28.685376 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:28.684661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.3:54041)
0927 21:31:28.685133 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:28.685212 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":262,"threads_started":1}
I20260927 21:31:28.764201 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:28.763531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.1:60139)
0927 21:31:28.763952 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:28.764027 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":224,"threads_started":1}
I20260927 21:31:29.245728 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:29.244939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.2:58001)
0927 21:31:29.245377 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:29.245474 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":221,"threads_started":1}
I20260927 21:31:29.687032 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:29.686315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.3:36211)
0927 21:31:29.686777 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:29.686855 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":252,"threads_started":1}
I20260927 21:31:29.766018 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:29.765224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.1:37629)
0927 21:31:29.765776 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:29.765857 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":313,"threads_started":1}
I20260927 21:31:30.247380 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:30.246641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.2:50665)
0927 21:31:30.247099 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:30.247182 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":232,"threads_started":1}
I20260927 21:31:30.688832 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:30.688002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.3:52823)
0927 21:31:30.688526 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:30.688643 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":275,"threads_started":1}
I20260927 21:31:30.767884 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:30.767004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.1:58281)
0927 21:31:30.767503 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:30.767597 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":264,"threads_started":1}
I20260927 21:31:31.249162 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:31.248358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.2:48613)
0927 21:31:31.248832 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:31.248925 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":212,"threads_started":1}
I20260927 21:31:31.690627 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:31.689886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.3:41517)
0927 21:31:31.690359 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:31.690451 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":222,"threads_started":1}
I20260927 21:31:31.769918 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:31.769016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.1:37739)
0927 21:31:31.769472 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:31.769648 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":250,"threads_started":1}
I20260927 21:31:32.250856 17519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:32.250161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.2:56493)
0927 21:31:32.250585 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:32.250663 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":213,"threads_started":1}
I20260927 21:31:32.692234 17520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:32.691535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.3:50607)
0927 21:31:32.691980 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:32.692057 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":232,"threads_started":1}
I20260927 21:31:32.771849 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:32.771066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.1:42563)
0927 21:31:32.771565 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:32.771668 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":250,"threads_started":1}
I20260927 21:31:33.252827 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:33.252052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.2:34421)
0927 21:31:33.252539 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:33.252646 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":245,"threads_started":1}
I20260927 21:31:33.675120 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 17045
I20260927 21:31:33.694065 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:33.676662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39148 (local address 127.0.97.1:37393)
0927 21:31:33.677296 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:33.677311 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:33.677350 (+ 39us) server_negotiation.cc:407] Connection header received
0927 21:31:33.678992 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:33.679002 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:33.679426 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:33.679781 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:33.681289 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:33.682580 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:33.683854 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:33.684522 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:33.684759 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:33.687489 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:33.687497 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:33.687513 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:33.687558 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:33.690724 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:33.691464 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:33.691471 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:33.691478 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:33.691650 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:33.692044 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:33.692055 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:33.692057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:33.692586 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:33.692914 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:33.693317 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 21:31:33.693755 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":329,"threads_started":1}
I20260927 21:31:33.694373 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:33.693484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46129 (local address 127.0.97.3:60461)
0927 21:31:33.694037 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:33.694165 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":325,"threads_started":1}
I20260927 21:31:33.695467 17208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:33.782137 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 17045
I20260927 21:31:33.799365 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 17216
I20260927 21:31:33.818886 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:33.800388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53214 (local address 127.0.97.2:44853)
0927 21:31:33.800959 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:33.800983 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:33.801022 (+ 39us) server_negotiation.cc:407] Connection header received
0927 21:31:33.802809 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:33.802819 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:33.803268 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:33.803649 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:33.805183 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:33.806539 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:33.807845 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:33.808529 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:33.808742 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:33.811791 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:33.811805 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:33.811823 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:33.811864 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:33.815099 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:33.815868 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:33.815876 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:33.815883 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:33.816078 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:33.816544 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:33.816550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:33.816560 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:33.817269 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:33.817832 (+ 563us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:33.818157 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 21:31:33.818587 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":295,"threads_started":1}
I20260927 21:31:33.820252 17339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:33.905517 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 17216
I20260927 21:31:33.924892 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 17347
I20260927 21:31:33.943323 17529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:33.925943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55040 (local address 127.0.97.3:33005)
0927 21:31:33.926529 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:33.926543 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:33.926586 (+ 43us) server_negotiation.cc:407] Connection header received
0927 21:31:33.928240 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:33.928249 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:33.928678 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:33.928965 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:33.930542 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:33.931785 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:33.933218 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:33.933947 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:33.934102 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:33.936805 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:33.936813 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:33.936828 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:33.936872 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:33.939861 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:33.940600 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:33.940607 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:33.940613 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:33.940780 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:33.941181 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:33.941185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:33.941187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:33.941847 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:33.942083 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:33.942553 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 21:31:33.943004 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":308,"threads_started":1}
I20260927 21:31:33.944777 17470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:34.028607 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 17347
2026-09-27T21:31:34Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64399 ms)
[----------] 30 tests from SecurityITest (303234 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:31:34 dist-test-slave-bxm0 krb5kdc[17533](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:31:34 dist-test-slave-bxm0 krb5kdc[17533](info): set up 2 sockets
Sep 27 21:31:34 dist-test-slave-bxm0 krb5kdc[17533](info): commencing operation
krb5kdc: starting...
W20260927 21:31:35.141599 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.009s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:31:35 dist-test-slave-bxm0 krb5kdc[17533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:31:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:31:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:31:35.396762 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40999
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:31:35.734732 17546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:35.735050 17546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:35.735232 17546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:35.742739 17546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:31:35.742851 17546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:35.742957 17546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:35.743074 17546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:31:35.743155 17546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:31:35.753881 17546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:35.755290 17546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:35.756628 17546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:35.769044 17551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:35.769090 17554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:35.769044 17552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:35.769793 17546 server_base.cc:1034] running on GCE node
I20260927 21:31:35.770570 17546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:35.771970 17546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:35.773195 17546 hybrid_clock.cc:648] HybridClock initialized: now 1790544695773156 us; error 41 us; skew 500 ppm
I20260927 21:31:35.773501 17546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:31:35 dist-test-slave-bxm0 krb5kdc[17533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:31:35.782986 17546 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:31:35.784695 17546 webserver.cc:460] Webserver started at http://127.0.97.62:43877/ using document root <none> and password file <none>
I20260927 21:31:35.785437 17546 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:35.785646 17546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:35.786002 17546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:31:35.789078 17546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "17c11f1be8d147e8a0d7cd82530f5b9f"
format_stamp: "Formatted at 2026-09-27 21:31:35 on dist-test-slave-bxm0"
server_key: "9fc983a83105abcb50655a0a8e7dfd63"
server_key_iv: "442f8a0bb0ff07b26b0b7ff63a904103"
server_key_version: "encryptionkey@0"
I20260927 21:31:35.789927 17546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "17c11f1be8d147e8a0d7cd82530f5b9f"
format_stamp: "Formatted at 2026-09-27 21:31:35 on dist-test-slave-bxm0"
server_key: "9fc983a83105abcb50655a0a8e7dfd63"
server_key_iv: "442f8a0bb0ff07b26b0b7ff63a904103"
server_key_version: "encryptionkey@0"
I20260927 21:31:35.795230 17546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 21:31:35.798791 17561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:35.800135 17546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:31:35.800330 17546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "17c11f1be8d147e8a0d7cd82530f5b9f"
format_stamp: "Formatted at 2026-09-27 21:31:35 on dist-test-slave-bxm0"
server_key: "9fc983a83105abcb50655a0a8e7dfd63"
server_key_iv: "442f8a0bb0ff07b26b0b7ff63a904103"
server_key_version: "encryptionkey@0"
I20260927 21:31:35.800563 17546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:35.817227 17546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:35.821586 17546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:35.821970 17546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:35.840662 17546 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40999
I20260927 21:31:35.840744 17612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40999 every 8 connection(s)
I20260927 21:31:35.842192 17546 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:31:35.846036 17613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:31:35.851397 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 17546
I20260927 21:31:35.851616 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:31:35.852447 388 external_mini_cluster.cc:1428] Setting key b5e3a9821b2f81e17a4f7020a457d749
I20260927 21:31:35.858004 17613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17c11f1be8d147e8a0d7cd82530f5b9f: Bootstrap starting.
I20260927 21:31:35.862043 17613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17c11f1be8d147e8a0d7cd82530f5b9f: Neither blocks nor log segments found. Creating new log.
I20260927 21:31:35.863415 17613 log.cc:824] T 00000000000000000000000000000000 P 17c11f1be8d147e8a0d7cd82530f5b9f: Log is configured to *not* fsync() on all Append() calls
I20260927 21:31:35.867401 17613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17c11f1be8d147e8a0d7cd82530f5b9f: No bootstrap required, opened a new log
Sep 27 21:31:35 dist-test-slave-bxm0 krb5kdc[17533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:31:35.875669 17613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17c11f1be8d147e8a0d7cd82530f5b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c11f1be8d147e8a0d7cd82530f5b9f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40999 } }
I20260927 21:31:35.876097 17613 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17c11f1be8d147e8a0d7cd82530f5b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:31:35.876232 17613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17c11f1be8d147e8a0d7cd82530f5b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c11f1be8d147e8a0d7cd82530f5b9f, State: Initialized, Role: FOLLOWER
I20260927 21:31:35.876996 17613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17c11f1be8d147e8a0d7cd82530f5b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c11f1be8d147e8a0d7cd82530f5b9f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40999 } }
I20260927 21:31:35.877311 17613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17c11f1be8d147e8a0d7cd82530f5b9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:31:35.877486 17613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17c11f1be8d147e8a0d7cd82530f5b9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:31:35.877787 17613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17c11f1be8d147e8a0d7cd82530f5b9f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:31:35.878389 17616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:35.854609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41894 (local address 127.0.97.62:40999)
0927 21:31:35.855531 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:35.855544 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:35.855589 (+ 45us) server_negotiation.cc:407] Connection header received
0927 21:31:35.857572 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:35.857596 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:35.858111 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:35.858468 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:35.860377 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:35.861806 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:35.863859 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:35.865147 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:35.865292 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:35.870483 (+ 5191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:35.870500 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:35.870520 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:35.870562 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:35.873917 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:35.874688 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:35.874696 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:35.874702 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:35.874900 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:35.875292 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:35.875298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:35.875301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:35.875861 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:35.876189 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:35.876626 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 21:31:35.877060 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":444,"threads_started":1}
I20260927 21:31:35.880294 17613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17c11f1be8d147e8a0d7cd82530f5b9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c11f1be8d147e8a0d7cd82530f5b9f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40999 } }
I20260927 21:31:35.880832 17613 leader_election.cc:302] T 00000000000000000000000000000000 P 17c11f1be8d147e8a0d7cd82530f5b9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17c11f1be8d147e8a0d7cd82530f5b9f; no voters:
I20260927 21:31:35.881590 17613 leader_election.cc:288] T 00000000000000000000000000000000 P 17c11f1be8d147e8a0d7cd82530f5b9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:31:35.881709 17618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17c11f1be8d147e8a0d7cd82530f5b9f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:31:35.882496 17618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17c11f1be8d147e8a0d7cd82530f5b9f [term 1 LEADER]: Becoming Leader. State: Replica: 17c11f1be8d147e8a0d7cd82530f5b9f, State: Running, Role: LEADER
I20260927 21:31:35.883163 17618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17c11f1be8d147e8a0d7cd82530f5b9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c11f1be8d147e8a0d7cd82530f5b9f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40999 } }
I20260927 21:31:35.884083 17613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17c11f1be8d147e8a0d7cd82530f5b9f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:31:35.886117 17619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17c11f1be8d147e8a0d7cd82530f5b9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17c11f1be8d147e8a0d7cd82530f5b9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c11f1be8d147e8a0d7cd82530f5b9f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40999 } } }
I20260927 21:31:35.886452 17619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17c11f1be8d147e8a0d7cd82530f5b9f [sys.catalog]: This master's current role is: LEADER
I20260927 21:31:35.887307 17625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:31:35.887147 17620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17c11f1be8d147e8a0d7cd82530f5b9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17c11f1be8d147e8a0d7cd82530f5b9f. Latest consensus state: current_term: 1 leader_uuid: "17c11f1be8d147e8a0d7cd82530f5b9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c11f1be8d147e8a0d7cd82530f5b9f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40999 } } }
I20260927 21:31:35.887532 17620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17c11f1be8d147e8a0d7cd82530f5b9f [sys.catalog]: This master's current role is: LEADER
I20260927 21:31:35.892818 17625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:31:35.902644 17625 catalog_manager.cc:1347] Generated new cluster ID: ff5336aa6f4b4e90a8735c4b73d49650
I20260927 21:31:35.902792 17625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:31:35.913118 17625 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:31:35.914515 17625 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:31:35.922101 17625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17c11f1be8d147e8a0d7cd82530f5b9f: Generated new TSK 0
I20260927 21:31:35.922984 17625 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 21:31:35.986181 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40999
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:31:36.329663 17639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:36.329959 17639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:36.330076 17639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:36.337280 17639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:36.337396 17639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:36.337510 17639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:31:36.347684 17639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:36.348974 17639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:36.350332 17639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:36.363340 17644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:36.363266 17645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:36.363315 17647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:36.364136 17639 server_base.cc:1034] running on GCE node
I20260927 21:31:36.364742 17639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:36.365734 17639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:36.366959 17639 hybrid_clock.cc:648] HybridClock initialized: now 1790544696366902 us; error 51 us; skew 500 ppm
I20260927 21:31:36.367295 17639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:31:36 dist-test-slave-bxm0 krb5kdc[17533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:31:36.370505 17639 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 21:31:36.371922 17639 webserver.cc:460] Webserver started at http://127.0.97.1:36025/ using document root <none> and password file <none>
I20260927 21:31:36.372619 17639 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:36.372764 17639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:36.373096 17639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:31:36.376113 17639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-0/data/instance:
uuid: "ed128a6a7fbb41bea5c90e282a861de0"
format_stamp: "Formatted at 2026-09-27 21:31:36 on dist-test-slave-bxm0"
server_key: "7b955ad9aed8882c344688538bf89bb3"
server_key_iv: "617aff1e2fab5ab62cfedf66a3e4c57f"
server_key_version: "encryptionkey@0"
I20260927 21:31:36.376899 17639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ed128a6a7fbb41bea5c90e282a861de0"
format_stamp: "Formatted at 2026-09-27 21:31:36 on dist-test-slave-bxm0"
server_key: "7b955ad9aed8882c344688538bf89bb3"
server_key_iv: "617aff1e2fab5ab62cfedf66a3e4c57f"
server_key_version: "encryptionkey@0"
I20260927 21:31:36.382215 17639 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 21:31:36.386046 17654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:36.387403 17639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:31:36.387593 17639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "ed128a6a7fbb41bea5c90e282a861de0"
format_stamp: "Formatted at 2026-09-27 21:31:36 on dist-test-slave-bxm0"
server_key: "7b955ad9aed8882c344688538bf89bb3"
server_key_iv: "617aff1e2fab5ab62cfedf66a3e4c57f"
server_key_version: "encryptionkey@0"
I20260927 21:31:36.387804 17639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:36.421693 17639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:36.425966 17639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:36.426290 17639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:36.427242 17639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:31:36.428617 17639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:31:36.428722 17639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:36.428875 17639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:31:36.428949 17639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:36.462447 17639 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45291
I20260927 21:31:36.462512 17766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45291 every 8 connection(s)
I20260927 21:31:36.463742 17639 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:31:36.471345 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 17639
I20260927 21:31:36.471545 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-0/wal/instance
I20260927 21:31:36.472198 388 external_mini_cluster.cc:1428] Setting key 51bf70f384f2a2061e6ca279a1d2b199
Sep 27 21:31:36 dist-test-slave-bxm0 krb5kdc[17533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:31:36.483852 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:36.466338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48897 (local address 127.0.97.62:40999)
0927 21:31:36.466814 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:36.466819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:36.467618 (+ 799us) server_negotiation.cc:407] Connection header received
0927 21:31:36.469475 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:36.469480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:36.469587 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:36.469756 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:36.471253 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:36.472113 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:36.473769 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:36.474184 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:36.474325 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:36.478063 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:36.478069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:36.478072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:36.478103 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:36.480676 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:36.481417 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:36.481426 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:36.481430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:36.481571 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:36.482056 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:36.482061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:36.482065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:36.482340 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:36.482514 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:36.483233 (+ 719us) server_negotiation.cc:299] Negotiation successful
0927 21:31:36.483471 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":257,"threads_started":1}
I20260927 21:31:36.485021 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:36.466756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40999 (local address 127.0.97.1:48897)
0927 21:31:36.467448 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:36.467497 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:36.469197 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:36.469994 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:36.470002 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:36.470538 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:31:36.471076 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:36.471092 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:36.472311 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:36.472318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:36.473564 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:36.473576 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:36.474421 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:36.474430 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:36.474677 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:36.475372 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:31:36.475397 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:31:36.477823 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:31:36.480831 (+ 3008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:31:36.480840 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:31:36.480858 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:31:36.481291 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:31:36.481733 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:31:36.481738 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:31:36.481742 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:31:36.481934 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:31:36.482506 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:31:36.482514 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:31:36.482855 (+ 341us) client_negotiation.cc:769] Sending connection context
0927 21:31:36.483127 (+ 272us) client_negotiation.cc:240] Negotiation successful
0927 21:31:36.483699 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":276,"threads_started":1}
I20260927 21:31:36.486748 17767 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40999
I20260927 21:31:36.487078 17767 heartbeater.cc:464] Registering TS with master...
I20260927 21:31:36.487959 17767 heartbeater.cc:511] Master 127.0.97.62:40999 requested a full tablet report, sending...
I20260927 21:31:36.490484 17578 ts_manager.cc:194] Registered new tserver with Master: ed128a6a7fbb41bea5c90e282a861de0 (127.0.97.1:45291)
I20260927 21:31:36.492774 17578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48897
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 21:31:36.528378 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40999
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:31:36.854192 17773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:36.854456 17773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:36.854619 17773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:36.862471 17773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:36.862624 17773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:36.862815 17773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:31:36.872961 17773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:36.874343 17773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:36.875664 17773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:36.889249 17781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:36.889353 17779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:36.889279 17778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:36.895560 17773 server_base.cc:1034] running on GCE node
I20260927 21:31:36.896266 17773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:36.897267 17773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:36.898535 17773 hybrid_clock.cc:648] HybridClock initialized: now 1790544696898474 us; error 42 us; skew 500 ppm
I20260927 21:31:36.898885 17773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:31:36 dist-test-slave-bxm0 krb5kdc[17533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:31:36.902266 17773 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 21:31:36.903771 17773 webserver.cc:460] Webserver started at http://127.0.97.2:41501/ using document root <none> and password file <none>
I20260927 21:31:36.904475 17773 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:36.904626 17773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:36.904894 17773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:31:36.907835 17773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-1/data/instance:
uuid: "472a05d8beda4a2ab2724f901056d1ff"
format_stamp: "Formatted at 2026-09-27 21:31:36 on dist-test-slave-bxm0"
server_key: "e1fff5a64bb128037c80ad4b00430c9b"
server_key_iv: "3ab1642bc44e42865a595c8316f35ddc"
server_key_version: "encryptionkey@0"
I20260927 21:31:36.908588 17773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-1/wal/instance:
uuid: "472a05d8beda4a2ab2724f901056d1ff"
format_stamp: "Formatted at 2026-09-27 21:31:36 on dist-test-slave-bxm0"
server_key: "e1fff5a64bb128037c80ad4b00430c9b"
server_key_iv: "3ab1642bc44e42865a595c8316f35ddc"
server_key_version: "encryptionkey@0"
I20260927 21:31:36.913558 17773 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 21:31:36.916833 17788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:36.918179 17773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:31:36.918358 17773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "472a05d8beda4a2ab2724f901056d1ff"
format_stamp: "Formatted at 2026-09-27 21:31:36 on dist-test-slave-bxm0"
server_key: "e1fff5a64bb128037c80ad4b00430c9b"
server_key_iv: "3ab1642bc44e42865a595c8316f35ddc"
server_key_version: "encryptionkey@0"
I20260927 21:31:36.918562 17773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:36.944396 17773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:36.948935 17773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:36.949237 17773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:36.950237 17773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:31:36.951601 17773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:31:36.951673 17773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:36.951819 17773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:31:36.951879 17773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:36.985002 17773 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40869
I20260927 21:31:36.985025 17900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40869 every 8 connection(s)
I20260927 21:31:36.986351 17773 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:31:36.992463 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 17773
I20260927 21:31:36.992681 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-1/wal/instance
I20260927 21:31:36.993049 388 external_mini_cluster.cc:1428] Setting key cbd5df8c619b022956aa87612a6926b1
Sep 27 21:31:36 dist-test-slave-bxm0 krb5kdc[17533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:31:37.006386 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:36.988646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53107 (local address 127.0.97.62:40999)
0927 21:31:36.989110 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:36.989115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:36.989971 (+ 856us) server_negotiation.cc:407] Connection header received
0927 21:31:36.992053 (+ 2082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:36.992059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:36.992158 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:36.992279 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:36.993965 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:36.994710 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:36.996265 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:36.996692 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:36.996826 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:37.000255 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:37.000260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:37.000263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:37.000296 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:37.002969 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:37.003913 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:37.003917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:37.003919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:37.004064 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:37.004519 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:37.004530 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:37.004533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:37.004870 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:37.005020 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:37.005756 (+ 736us) server_negotiation.cc:299] Negotiation successful
0927 21:31:37.005986 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":364,"thread_start_us":257,"threads_started":1}
I20260927 21:31:37.007414 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:36.989078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40999 (local address 127.0.97.2:53107)
0927 21:31:36.989802 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:36.989845 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:36.991754 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:36.992514 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:36.992522 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:36.993078 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:31:36.993782 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:36.993811 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:36.994863 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:36.994869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:36.996107 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:36.996119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:36.996913 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:36.996920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:36.997153 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:36.997864 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:31:36.997886 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:31:37.000017 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:31:37.003116 (+ 3099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:31:37.003128 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:31:37.003148 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:31:37.003772 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:31:37.004189 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:31:37.004194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:31:37.004198 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:31:37.004392 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:31:37.005014 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:31:37.005022 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:31:37.005336 (+ 314us) client_negotiation.cc:769] Sending connection context
0927 21:31:37.005667 (+ 331us) client_negotiation.cc:240] Negotiation successful
0927 21:31:37.006224 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":269,"threads_started":1}
I20260927 21:31:37.008874 17901 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40999
I20260927 21:31:37.009212 17901 heartbeater.cc:464] Registering TS with master...
I20260927 21:31:37.010131 17901 heartbeater.cc:511] Master 127.0.97.62:40999 requested a full tablet report, sending...
I20260927 21:31:37.011746 17578 ts_manager.cc:194] Registered new tserver with Master: 472a05d8beda4a2ab2724f901056d1ff (127.0.97.2:40869)
I20260927 21:31:37.012707 17578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53107
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 21:31:37.048712 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40999
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:31:37.388538 17907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:37.388917 17907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:37.389106 17907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:37.396968 17907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:37.397192 17907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:37.397403 17907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:31:37.408766 17907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:37.410277 17907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:37.411675 17907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:37.425689 17912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:37.425648 17913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:37.425773 17915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:37.426482 17907 server_base.cc:1034] running on GCE node
I20260927 21:31:37.427057 17907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:37.427980 17907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:37.429181 17907 hybrid_clock.cc:648] HybridClock initialized: now 1790544697429138 us; error 33 us; skew 500 ppm
I20260927 21:31:37.429502 17907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:31:37 dist-test-slave-bxm0 krb5kdc[17533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:31:37.432528 17907 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 21:31:37.433960 17907 webserver.cc:460] Webserver started at http://127.0.97.3:39871/ using document root <none> and password file <none>
I20260927 21:31:37.434623 17907 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:37.434762 17907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:37.435066 17907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:31:37.437968 17907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-2/data/instance:
uuid: "e913b09d0d194f65b4f301cc96a6f607"
format_stamp: "Formatted at 2026-09-27 21:31:37 on dist-test-slave-bxm0"
server_key: "8f606aa69ba8f4576936b144abda2d29"
server_key_iv: "b1d43fe0a49471d332246754e7f7855b"
server_key_version: "encryptionkey@0"
I20260927 21:31:37.438738 17907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e913b09d0d194f65b4f301cc96a6f607"
format_stamp: "Formatted at 2026-09-27 21:31:37 on dist-test-slave-bxm0"
server_key: "8f606aa69ba8f4576936b144abda2d29"
server_key_iv: "b1d43fe0a49471d332246754e7f7855b"
server_key_version: "encryptionkey@0"
I20260927 21:31:37.443722 17907 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:31:37.447067 17922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:37.448364 17907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:31:37.448549 17907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "e913b09d0d194f65b4f301cc96a6f607"
format_stamp: "Formatted at 2026-09-27 21:31:37 on dist-test-slave-bxm0"
server_key: "8f606aa69ba8f4576936b144abda2d29"
server_key_iv: "b1d43fe0a49471d332246754e7f7855b"
server_key_version: "encryptionkey@0"
I20260927 21:31:37.448777 17907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:37.472100 17907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:37.476231 17907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:37.476545 17907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:37.477506 17907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:31:37.478856 17907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:31:37.478967 17907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:37.479112 17907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:31:37.479184 17907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:37.496042 17767 heartbeater.cc:503] Master 127.0.97.62:40999 was elected leader, sending a full tablet report...
I20260927 21:31:37.514207 17907 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46553
I20260927 21:31:37.514297 18034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46553 every 8 connection(s)
I20260927 21:31:37.515554 17907 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:31:37.522624 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 17907
I20260927 21:31:37.522815 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-2/wal/instance
I20260927 21:31:37.523335 388 external_mini_cluster.cc:1428] Setting key a54a408cb182de7d431c9b6e81f00703
Sep 27 21:31:37 dist-test-slave-bxm0 krb5kdc[17533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:31:37.535421 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:37.518137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58923 (local address 127.0.97.62:40999)
0927 21:31:37.518612 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:37.518617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:37.519438 (+ 821us) server_negotiation.cc:407] Connection header received
0927 21:31:37.521271 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:37.521278 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:37.521385 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:37.521522 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:37.523459 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.524495 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.526569 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.526948 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.527045 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:37.530322 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:37.530326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:37.530329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:37.530360 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:37.532767 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:37.533430 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:37.533434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:37.533436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:37.533545 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:37.533952 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:37.533955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:37.533957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:37.534210 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:37.534327 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:37.534953 (+ 626us) server_negotiation.cc:299] Negotiation successful
0927 21:31:37.535124 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":255,"threads_started":1}
I20260927 21:31:37.536383 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:37.518571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40999 (local address 127.0.97.3:58923)
0927 21:31:37.519264 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:37.519302 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:37.520999 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:37.521782 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:37.521791 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:37.522434 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:31:37.523241 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:37.523266 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.524658 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.524663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:37.526417 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:37.526433 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.527127 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.527131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:37.527318 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:37.528006 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:31:37.528027 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:31:37.530138 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:31:37.532889 (+ 2751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:31:37.532897 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:31:37.532911 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:31:37.533324 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:31:37.533682 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:31:37.533686 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:31:37.533688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:31:37.533852 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:31:37.534311 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:31:37.534316 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:31:37.534634 (+ 318us) client_negotiation.cc:769] Sending connection context
0927 21:31:37.534855 (+ 221us) client_negotiation.cc:240] Negotiation successful
0927 21:31:37.535241 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":282,"threads_started":1}
I20260927 21:31:37.537897 18035 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40999
I20260927 21:31:37.538213 18035 heartbeater.cc:464] Registering TS with master...
I20260927 21:31:37.538897 18035 heartbeater.cc:511] Master 127.0.97.62:40999 requested a full tablet report, sending...
I20260927 21:31:37.540275 17577 ts_manager.cc:194] Registered new tserver with Master: e913b09d0d194f65b4f301cc96a6f607 (127.0.97.3:46553)
I20260927 21:31:37.541169 17577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58923
I20260927 21:31:37.551164 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:31:37.568841 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:37.555767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41904 (local address 127.0.97.62:40999)
0927 21:31:37.555967 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:37.555976 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:37.556405 (+ 429us) server_negotiation.cc:407] Connection header received
0927 21:31:37.556594 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:37.556600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:37.556678 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:37.556783 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:37.558247 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.558994 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.560500 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.560924 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.561058 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:37.563569 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:37.563575 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:37.563580 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:37.563621 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:37.566116 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:37.566823 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:37.566829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:37.566832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:37.566966 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:37.567333 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:37.567338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:37.567342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:37.567672 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:37.567933 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:37.568263 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 21:31:37.568517 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:31:37.572747 17577 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:31:37.579559 17577 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:31:37.614768 18051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:37.592411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49070 (local address 127.0.97.1:45291)
0927 21:31:37.606152 (+ 13741us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:37.606163 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:37.606193 (+ 30us) server_negotiation.cc:407] Connection header received
0927 21:31:37.606293 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:37.606300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:37.606600 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:37.606857 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:37.607532 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.609203 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.612341 (+ 3138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.613343 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.613509 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:37.613776 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:37.614263 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 21:31:37.614442 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13588,"thread_start_us":13429,"threads_started":1}
I20260927 21:31:37.614712 18048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:37.592308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45291 (local address 127.0.0.1:49070)
0927 21:31:37.605432 (+ 13124us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:37.605451 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:37.605564 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:37.606857 (+ 1293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:37.606864 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:37.606901 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:37.607331 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:37.607345 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.609349 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.609355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:37.612133 (+ 2778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:37.612148 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.613538 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.613544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:37.613764 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:37.613820 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 21:31:37.613996 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 21:31:37.614221 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12991,"mutex_wait_us":34,"thread_start_us":13386,"threads_started":1}
I20260927 21:31:37.615567 18049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:37.592107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42522 (local address 127.0.97.3:46553)
0927 21:31:37.601826 (+ 9719us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:37.601841 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:37.605376 (+ 3535us) server_negotiation.cc:407] Connection header received
0927 21:31:37.607283 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:37.607293 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:37.607536 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:37.607731 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:37.608374 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.609971 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.612453 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.613804 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.614366 (+ 562us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:37.614582 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:37.614649 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 21:31:37.614810 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9427,"thread_start_us":9232,"threads_started":1}
I20260927 21:31:37.615630 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:37.606342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57240 (local address 127.0.97.2:40869)
0927 21:31:37.607614 (+ 1272us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:37.607621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:37.607640 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:31:37.607748 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:37.607755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:37.607972 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:37.608173 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:37.608761 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.610266 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.612275 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.613792 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.613950 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:37.614145 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:37.615093 (+ 948us) server_negotiation.cc:299] Negotiation successful
0927 21:31:37.615287 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1125,"thread_start_us":1012,"threads_started":1}
I20260927 21:31:37.616106 18050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:37.591954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46553 (local address 127.0.0.1:42522)
0927 21:31:37.605174 (+ 13220us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:37.605241 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:37.605484 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:37.607733 (+ 2249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:37.607737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:37.607762 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:37.608128 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:37.608142 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.610119 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.610124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:37.612310 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:37.612322 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.614077 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.614084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:37.614333 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:37.614375 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 21:31:37.614458 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 21:31:37.614589 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12851,"mutex_wait_us":18,"thread_start_us":13186,"threads_started":1}
I20260927 21:31:37.616607 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:37.606094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40869 (local address 127.0.0.1:57240)
0927 21:31:37.606668 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:37.606681 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:37.606800 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:37.608178 (+ 1378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:37.608186 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:37.608212 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:37.608585 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:37.608598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.610403 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.610412 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:37.612118 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:37.612131 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.614046 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.614055 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:37.614257 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:37.614756 (+ 499us) client_negotiation.cc:769] Sending connection context
0927 21:31:37.614844 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 21:31:37.614985 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":99,"thread_start_us":320,"threads_started":1}
I20260927 21:31:37.619793 17836 tablet_service.cc:1467] Processing CreateTablet for tablet 35d8c62cd0b546d99f0ec117d722589e (DEFAULT_TABLE table=test-table [id=76724951ac7348f6af7d23c88a536e5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:31:37.619699 17702 tablet_service.cc:1467] Processing CreateTablet for tablet 35d8c62cd0b546d99f0ec117d722589e (DEFAULT_TABLE table=test-table [id=76724951ac7348f6af7d23c88a536e5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:31:37.619701 17970 tablet_service.cc:1467] Processing CreateTablet for tablet 35d8c62cd0b546d99f0ec117d722589e (DEFAULT_TABLE table=test-table [id=76724951ac7348f6af7d23c88a536e5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:31:37.620985 17836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35d8c62cd0b546d99f0ec117d722589e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:31:37.621091 17970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35d8c62cd0b546d99f0ec117d722589e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:31:37.625954 17702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35d8c62cd0b546d99f0ec117d722589e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:31:37.634856 18054 tablet_bootstrap.cc:492] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607: Bootstrap starting.
I20260927 21:31:37.635784 18055 tablet_bootstrap.cc:492] T 35d8c62cd0b546d99f0ec117d722589e P 472a05d8beda4a2ab2724f901056d1ff: Bootstrap starting.
I20260927 21:31:37.638721 18054 tablet_bootstrap.cc:654] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607: Neither blocks nor log segments found. Creating new log.
I20260927 21:31:37.639374 18055 tablet_bootstrap.cc:654] T 35d8c62cd0b546d99f0ec117d722589e P 472a05d8beda4a2ab2724f901056d1ff: Neither blocks nor log segments found. Creating new log.
I20260927 21:31:37.640076 18054 log.cc:824] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607: Log is configured to *not* fsync() on all Append() calls
I20260927 21:31:37.640636 18055 log.cc:824] T 35d8c62cd0b546d99f0ec117d722589e P 472a05d8beda4a2ab2724f901056d1ff: Log is configured to *not* fsync() on all Append() calls
I20260927 21:31:37.642249 18057 tablet_bootstrap.cc:492] T 35d8c62cd0b546d99f0ec117d722589e P ed128a6a7fbb41bea5c90e282a861de0: Bootstrap starting.
I20260927 21:31:37.644114 18054 tablet_bootstrap.cc:492] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607: No bootstrap required, opened a new log
I20260927 21:31:37.644403 18054 ts_tablet_manager.cc:1397] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260927 21:31:37.645498 18055 tablet_bootstrap.cc:492] T 35d8c62cd0b546d99f0ec117d722589e P 472a05d8beda4a2ab2724f901056d1ff: No bootstrap required, opened a new log
I20260927 21:31:37.645830 18055 ts_tablet_manager.cc:1397] T 35d8c62cd0b546d99f0ec117d722589e P 472a05d8beda4a2ab2724f901056d1ff: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 21:31:37.645895 18057 tablet_bootstrap.cc:654] T 35d8c62cd0b546d99f0ec117d722589e P ed128a6a7fbb41bea5c90e282a861de0: Neither blocks nor log segments found. Creating new log.
I20260927 21:31:37.647096 18057 log.cc:824] T 35d8c62cd0b546d99f0ec117d722589e P ed128a6a7fbb41bea5c90e282a861de0: Log is configured to *not* fsync() on all Append() calls
I20260927 21:31:37.650257 18057 tablet_bootstrap.cc:492] T 35d8c62cd0b546d99f0ec117d722589e P ed128a6a7fbb41bea5c90e282a861de0: No bootstrap required, opened a new log
I20260927 21:31:37.650529 18057 ts_tablet_manager.cc:1397] T 35d8c62cd0b546d99f0ec117d722589e P ed128a6a7fbb41bea5c90e282a861de0: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260927 21:31:37.654703 18055 raft_consensus.cc:348] T 35d8c62cd0b546d99f0ec117d722589e P 472a05d8beda4a2ab2724f901056d1ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e913b09d0d194f65b4f301cc96a6f607" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46553 } } peers { permanent_uuid: "ed128a6a7fbb41bea5c90e282a861de0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45291 } } peers { permanent_uuid: "472a05d8beda4a2ab2724f901056d1ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40869 } }
I20260927 21:31:37.655161 18055 raft_consensus.cc:374] T 35d8c62cd0b546d99f0ec117d722589e P 472a05d8beda4a2ab2724f901056d1ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:31:37.655277 18055 raft_consensus.cc:729] T 35d8c62cd0b546d99f0ec117d722589e P 472a05d8beda4a2ab2724f901056d1ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 472a05d8beda4a2ab2724f901056d1ff, State: Initialized, Role: FOLLOWER
I20260927 21:31:37.655014 18054 raft_consensus.cc:348] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e913b09d0d194f65b4f301cc96a6f607" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46553 } } peers { permanent_uuid: "ed128a6a7fbb41bea5c90e282a861de0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45291 } } peers { permanent_uuid: "472a05d8beda4a2ab2724f901056d1ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40869 } }
I20260927 21:31:37.655444 18054 raft_consensus.cc:374] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:31:37.655544 18054 raft_consensus.cc:729] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e913b09d0d194f65b4f301cc96a6f607, State: Initialized, Role: FOLLOWER
I20260927 21:31:37.656008 18055 consensus_queue.cc:260] T 35d8c62cd0b546d99f0ec117d722589e P 472a05d8beda4a2ab2724f901056d1ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e913b09d0d194f65b4f301cc96a6f607" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46553 } } peers { permanent_uuid: "ed128a6a7fbb41bea5c90e282a861de0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45291 } } peers { permanent_uuid: "472a05d8beda4a2ab2724f901056d1ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40869 } }
I20260927 21:31:37.656215 18054 consensus_queue.cc:260] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e913b09d0d194f65b4f301cc96a6f607" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46553 } } peers { permanent_uuid: "ed128a6a7fbb41bea5c90e282a861de0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45291 } } peers { permanent_uuid: "472a05d8beda4a2ab2724f901056d1ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40869 } }
I20260927 21:31:37.660430 18035 heartbeater.cc:503] Master 127.0.97.62:40999 was elected leader, sending a full tablet report...
I20260927 21:31:37.661446 18054 ts_tablet_manager.cc:1428] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607: Time spent starting tablet: real 0.017s user 0.014s sys 0.001s
I20260927 21:31:37.662452 17901 heartbeater.cc:503] Master 127.0.97.62:40999 was elected leader, sending a full tablet report...
I20260927 21:31:37.662600 18057 raft_consensus.cc:348] T 35d8c62cd0b546d99f0ec117d722589e P ed128a6a7fbb41bea5c90e282a861de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e913b09d0d194f65b4f301cc96a6f607" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46553 } } peers { permanent_uuid: "ed128a6a7fbb41bea5c90e282a861de0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45291 } } peers { permanent_uuid: "472a05d8beda4a2ab2724f901056d1ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40869 } }
I20260927 21:31:37.663075 18057 raft_consensus.cc:374] T 35d8c62cd0b546d99f0ec117d722589e P ed128a6a7fbb41bea5c90e282a861de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:31:37.663169 18057 raft_consensus.cc:729] T 35d8c62cd0b546d99f0ec117d722589e P ed128a6a7fbb41bea5c90e282a861de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed128a6a7fbb41bea5c90e282a861de0, State: Initialized, Role: FOLLOWER
I20260927 21:31:37.663174 18055 ts_tablet_manager.cc:1428] T 35d8c62cd0b546d99f0ec117d722589e P 472a05d8beda4a2ab2724f901056d1ff: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260927 21:31:37.663873 18057 consensus_queue.cc:260] T 35d8c62cd0b546d99f0ec117d722589e P ed128a6a7fbb41bea5c90e282a861de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e913b09d0d194f65b4f301cc96a6f607" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46553 } } peers { permanent_uuid: "ed128a6a7fbb41bea5c90e282a861de0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45291 } } peers { permanent_uuid: "472a05d8beda4a2ab2724f901056d1ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40869 } }
I20260927 21:31:37.665706 18057 ts_tablet_manager.cc:1428] T 35d8c62cd0b546d99f0ec117d722589e P ed128a6a7fbb41bea5c90e282a861de0: Time spent starting tablet: real 0.015s user 0.014s sys 0.002s
W20260927 21:31:37.716346 17768 tablet.cc:2369] T 35d8c62cd0b546d99f0ec117d722589e P ed128a6a7fbb41bea5c90e282a861de0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:31:37.738306 17902 tablet.cc:2369] T 35d8c62cd0b546d99f0ec117d722589e P 472a05d8beda4a2ab2724f901056d1ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:31:37.767210 18036 tablet.cc:2369] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:31:37.887034 18061 raft_consensus.cc:482] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:31:37.887382 18061 raft_consensus.cc:504] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e913b09d0d194f65b4f301cc96a6f607" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46553 } } peers { permanent_uuid: "ed128a6a7fbb41bea5c90e282a861de0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45291 } } peers { permanent_uuid: "472a05d8beda4a2ab2724f901056d1ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40869 } }
I20260927 21:31:37.889176 18061 leader_election.cc:288] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed128a6a7fbb41bea5c90e282a861de0 (127.0.97.1:45291), 472a05d8beda4a2ab2724f901056d1ff (127.0.97.2:40869)
I20260927 21:31:37.896219 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:37.889806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43141 (local address 127.0.97.2:40869)
0927 21:31:37.890058 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:37.890063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:37.890078 (+ 15us) server_negotiation.cc:407] Connection header received
0927 21:31:37.890134 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:37.890137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:37.890202 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:37.890304 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:37.890772 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.891844 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.894299 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.895403 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.895540 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:37.895643 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:37.895862 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 21:31:37.895999 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260927 21:31:37.896246 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:37.889414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40869 (local address 127.0.97.3:43141)
0927 21:31:37.889639 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:37.889664 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:37.889853 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:37.890324 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:37.890328 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:37.890354 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:37.890589 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:37.890601 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.892048 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.892053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:37.894137 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:37.894152 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.895560 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.895564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:37.895693 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:37.895717 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 21:31:37.895790 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 21:31:37.895888 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":27,"spinlock_wait_cycles":896}
I20260927 21:31:37.896886 18051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:37.889701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37475 (local address 127.0.97.1:45291)
0927 21:31:37.889920 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:37.889928 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:37.890339 (+ 411us) server_negotiation.cc:407] Connection header received
0927 21:31:37.890572 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:37.890576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:37.890633 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:37.890740 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:37.891331 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.892337 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.894634 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.895829 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.895958 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:37.896050 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:37.896497 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 21:31:37.896628 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 21:31:37.896965 18063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:37.889572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45291 (local address 127.0.97.3:37475)
0927 21:31:37.890227 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:37.890241 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:37.890400 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:37.890768 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:37.890774 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:37.890824 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:37.891182 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:37.891191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.892481 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.892487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:37.894497 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:37.894506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.896065 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.896071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:37.896242 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:37.896287 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 21:31:37.896450 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 21:31:37.896576 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":1,"thread_start_us":812,"threads_started":1}
I20260927 21:31:37.897369 17856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d8c62cd0b546d99f0ec117d722589e" candidate_uuid: "e913b09d0d194f65b4f301cc96a6f607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "472a05d8beda4a2ab2724f901056d1ff" is_pre_election: true
I20260927 21:31:37.897903 17856 raft_consensus.cc:2454] T 35d8c62cd0b546d99f0ec117d722589e P 472a05d8beda4a2ab2724f901056d1ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e913b09d0d194f65b4f301cc96a6f607 in term 0.
I20260927 21:31:37.897993 17722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d8c62cd0b546d99f0ec117d722589e" candidate_uuid: "e913b09d0d194f65b4f301cc96a6f607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed128a6a7fbb41bea5c90e282a861de0" is_pre_election: true
I20260927 21:31:37.898509 17722 raft_consensus.cc:2454] T 35d8c62cd0b546d99f0ec117d722589e P ed128a6a7fbb41bea5c90e282a861de0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e913b09d0d194f65b4f301cc96a6f607 in term 0.
I20260927 21:31:37.898600 17925 leader_election.cc:302] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 472a05d8beda4a2ab2724f901056d1ff, e913b09d0d194f65b4f301cc96a6f607; no voters:
I20260927 21:31:37.899041 18061 raft_consensus.cc:2781] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:31:37.899178 18061 raft_consensus.cc:482] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:31:37.899308 18061 raft_consensus.cc:3037] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:31:37.901551 18061 raft_consensus.cc:504] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e913b09d0d194f65b4f301cc96a6f607" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46553 } } peers { permanent_uuid: "ed128a6a7fbb41bea5c90e282a861de0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45291 } } peers { permanent_uuid: "472a05d8beda4a2ab2724f901056d1ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40869 } }
I20260927 21:31:37.902272 18061 leader_election.cc:288] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607 [CANDIDATE]: Term 1 election: Requested vote from peers ed128a6a7fbb41bea5c90e282a861de0 (127.0.97.1:45291), 472a05d8beda4a2ab2724f901056d1ff (127.0.97.2:40869)
I20260927 21:31:37.902805 17856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d8c62cd0b546d99f0ec117d722589e" candidate_uuid: "e913b09d0d194f65b4f301cc96a6f607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "472a05d8beda4a2ab2724f901056d1ff"
I20260927 21:31:37.902853 17722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d8c62cd0b546d99f0ec117d722589e" candidate_uuid: "e913b09d0d194f65b4f301cc96a6f607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed128a6a7fbb41bea5c90e282a861de0"
I20260927 21:31:37.903024 17856 raft_consensus.cc:3037] T 35d8c62cd0b546d99f0ec117d722589e P 472a05d8beda4a2ab2724f901056d1ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:31:37.903066 17722 raft_consensus.cc:3037] T 35d8c62cd0b546d99f0ec117d722589e P ed128a6a7fbb41bea5c90e282a861de0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:31:37.905478 17856 raft_consensus.cc:2454] T 35d8c62cd0b546d99f0ec117d722589e P 472a05d8beda4a2ab2724f901056d1ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e913b09d0d194f65b4f301cc96a6f607 in term 1.
I20260927 21:31:37.905489 17722 raft_consensus.cc:2454] T 35d8c62cd0b546d99f0ec117d722589e P ed128a6a7fbb41bea5c90e282a861de0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e913b09d0d194f65b4f301cc96a6f607 in term 1.
I20260927 21:31:37.905934 17925 leader_election.cc:302] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 472a05d8beda4a2ab2724f901056d1ff, e913b09d0d194f65b4f301cc96a6f607; no voters:
I20260927 21:31:37.906263 18061 raft_consensus.cc:2781] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:31:37.906672 18061 raft_consensus.cc:686] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607 [term 1 LEADER]: Becoming Leader. State: Replica: e913b09d0d194f65b4f301cc96a6f607, State: Running, Role: LEADER
I20260927 21:31:37.907419 18061 consensus_queue.cc:237] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e913b09d0d194f65b4f301cc96a6f607" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46553 } } peers { permanent_uuid: "ed128a6a7fbb41bea5c90e282a861de0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45291 } } peers { permanent_uuid: "472a05d8beda4a2ab2724f901056d1ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40869 } }
I20260927 21:31:37.912297 17576 catalog_manager.cc:5495] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607 reported cstate change: term changed from 0 to 1, leader changed from <none> to e913b09d0d194f65b4f301cc96a6f607 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "e913b09d0d194f65b4f301cc96a6f607" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e913b09d0d194f65b4f301cc96a6f607" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed128a6a7fbb41bea5c90e282a861de0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "472a05d8beda4a2ab2724f901056d1ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40869 } health_report { overall_health: UNKNOWN } } }
I20260927 21:31:37.971237 18049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:37.965860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42534 (local address 127.0.97.3:46553)
0927 21:31:37.966086 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:37.966092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:37.966114 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:31:37.966240 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:37.966244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:37.966355 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:37.966531 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:31:37.966938 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.967676 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.969077 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:37.969501 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:37.969657 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:37.970148 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:31:37.970658 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:31:37.970830 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:37.970881 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 21:31:37.970964 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 21:31:37.979523 17722 raft_consensus.cc:1261] T 35d8c62cd0b546d99f0ec117d722589e P ed128a6a7fbb41bea5c90e282a861de0 [term 1 FOLLOWER]: Refusing update from remote peer e913b09d0d194f65b4f301cc96a6f607: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:31:37.979524 17856 raft_consensus.cc:1261] T 35d8c62cd0b546d99f0ec117d722589e P 472a05d8beda4a2ab2724f901056d1ff [term 1 FOLLOWER]: Refusing update from remote peer e913b09d0d194f65b4f301cc96a6f607: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:31:37.980655 18061 consensus_queue.cc:1035] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed128a6a7fbb41bea5c90e282a861de0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:31:37.980948 18064 consensus_queue.cc:1035] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "472a05d8beda4a2ab2724f901056d1ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:31:38.008919 18069 mvcc.cc:205] Tried to move back new op lower bound from 7334071082913026048 to 7334071082633625600. Current Snapshot: MvccSnapshot[applied={T|T < 7334071082913026048}]
I20260927 21:31:38.017674 17577 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:31:38.018013 17577 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:31:38.022261 17577 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17c11f1be8d147e8a0d7cd82530f5b9f: Sending DeleteTablet for 3 replicas of tablet 35d8c62cd0b546d99f0ec117d722589e
I20260927 21:31:38.023576 17702 tablet_service.cc:1514] Processing DeleteTablet for tablet 35d8c62cd0b546d99f0ec117d722589e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:31:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49070
I20260927 21:31:38.023604 17836 tablet_service.cc:1514] Processing DeleteTablet for tablet 35d8c62cd0b546d99f0ec117d722589e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:31:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57240
I20260927 21:31:38.024642 17970 tablet_service.cc:1514] Processing DeleteTablet for tablet 35d8c62cd0b546d99f0ec117d722589e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:31:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42522
I20260927 21:31:38.025743 18076 tablet_replica.cc:330] T 35d8c62cd0b546d99f0ec117d722589e P ed128a6a7fbb41bea5c90e282a861de0: stopping tablet replica
I20260927 21:31:38.026201 18076 raft_consensus.cc:2229] T 35d8c62cd0b546d99f0ec117d722589e P ed128a6a7fbb41bea5c90e282a861de0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:31:38.026736 18076 raft_consensus.cc:2258] T 35d8c62cd0b546d99f0ec117d722589e P ed128a6a7fbb41bea5c90e282a861de0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:31:38.027506 18077 tablet_replica.cc:330] T 35d8c62cd0b546d99f0ec117d722589e P 472a05d8beda4a2ab2724f901056d1ff: stopping tablet replica
I20260927 21:31:38.027932 18077 raft_consensus.cc:2229] T 35d8c62cd0b546d99f0ec117d722589e P 472a05d8beda4a2ab2724f901056d1ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:31:38.028017 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 17639
I20260927 21:31:38.028432 18077 raft_consensus.cc:2258] T 35d8c62cd0b546d99f0ec117d722589e P 472a05d8beda4a2ab2724f901056d1ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:31:38.032209 18077 ts_tablet_manager.cc:1905] T 35d8c62cd0b546d99f0ec117d722589e P 472a05d8beda4a2ab2724f901056d1ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:31:38.035989 18076 ts_tablet_manager.cc:1905] T 35d8c62cd0b546d99f0ec117d722589e P ed128a6a7fbb41bea5c90e282a861de0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:31:38.041213 18077 ts_tablet_manager.cc:1918] T 35d8c62cd0b546d99f0ec117d722589e P 472a05d8beda4a2ab2724f901056d1ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:31:38.041429 18077 log.cc:1196] T 35d8c62cd0b546d99f0ec117d722589e P 472a05d8beda4a2ab2724f901056d1ff: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-1/wal/wals/35d8c62cd0b546d99f0ec117d722589e
I20260927 21:31:38.042102 18077 ts_tablet_manager.cc:1939] T 35d8c62cd0b546d99f0ec117d722589e P 472a05d8beda4a2ab2724f901056d1ff: Deleting consensus metadata
I20260927 21:31:38.044238 18078 tablet_replica.cc:330] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607: stopping tablet replica
I20260927 21:31:38.044412 17564 catalog_manager.cc:4841] TS 472a05d8beda4a2ab2724f901056d1ff (127.0.97.2:40869): tablet 35d8c62cd0b546d99f0ec117d722589e (table test-table [id=76724951ac7348f6af7d23c88a536e5a]) successfully deleted
I20260927 21:31:38.044742 18078 raft_consensus.cc:2229] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:31:38.045596 18078 raft_consensus.cc:2258] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 21:31:38 dist-test-slave-bxm0 krb5kdc[17533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 21:31:38.048033 18078 ts_tablet_manager.cc:1905] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:31:38.051230 18076 ts_tablet_manager.cc:1918] T 35d8c62cd0b546d99f0ec117d722589e P ed128a6a7fbb41bea5c90e282a861de0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:31:38.051440 18076 log.cc:1196] T 35d8c62cd0b546d99f0ec117d722589e P ed128a6a7fbb41bea5c90e282a861de0: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-0/wal/wals/35d8c62cd0b546d99f0ec117d722589e
I20260927 21:31:38.052142 18076 ts_tablet_manager.cc:1939] T 35d8c62cd0b546d99f0ec117d722589e P ed128a6a7fbb41bea5c90e282a861de0: Deleting consensus metadata
I20260927 21:31:38.054548 17563 catalog_manager.cc:4841] TS ed128a6a7fbb41bea5c90e282a861de0 (127.0.97.1:45291): tablet 35d8c62cd0b546d99f0ec117d722589e (table test-table [id=76724951ac7348f6af7d23c88a536e5a]) successfully deleted
I20260927 21:31:38.055688 18078 ts_tablet_manager.cc:1918] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:31:38.055869 18078 log.cc:1196] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790544390748963-388-0/minicluster-data/ts-2/wal/wals/35d8c62cd0b546d99f0ec117d722589e
I20260927 21:31:38.056421 18078 ts_tablet_manager.cc:1939] T 35d8c62cd0b546d99f0ec117d722589e P e913b09d0d194f65b4f301cc96a6f607: Deleting consensus metadata
I20260927 21:31:38.057742 18051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:38.029213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49072 (local address 127.0.97.1:45291)
0927 21:31:38.029449 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:38.029458 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:38.035143 (+ 5685us) server_negotiation.cc:407] Connection header received
0927 21:31:38.035399 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:38.035407 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:38.035516 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:38.035624 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:38.037711 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:38.038889 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:38.041019 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:38.042039 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:38.042160 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:38.048876 (+ 6716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:38.048895 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:38.048919 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:38.048975 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:38.053726 (+ 4751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:38.055120 (+ 1394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:38.055138 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:38.055143 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:38.055304 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:38.055807 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:38.055816 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:38.055821 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:38.056271 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:38.056589 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:38.057007 (+ 418us) server_negotiation.cc:299] Negotiation successful
0927 21:31:38.057246 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 21:31:38.058151 17564 catalog_manager.cc:4841] TS e913b09d0d194f65b4f301cc96a6f607 (127.0.97.3:46553): tablet 35d8c62cd0b546d99f0ec117d722589e (table test-table [id=76724951ac7348f6af7d23c88a536e5a]) successfully deleted
I20260927 21:31:38.058447 17762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:38.153367 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 17639
I20260927 21:31:38.177865 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 17773
Sep 27 21:31:38 dist-test-slave-bxm0 krb5kdc[17533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 21:31:38.193689 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:38.178795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57242 (local address 127.0.97.2:40869)
0927 21:31:38.179027 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:38.179032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:38.179052 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:31:38.179220 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:38.179226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:38.179302 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:38.179420 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:38.180616 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:38.181348 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:38.182681 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:38.183098 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:38.183245 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:38.187563 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:38.187579 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:38.187595 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:38.187631 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:38.190661 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:38.191454 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:38.191462 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:38.191464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:38.191602 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:38.191998 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:38.192003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:38.192006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:38.192410 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:38.192720 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:38.193031 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 21:31:38.193256 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:31:38.194411 17896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:38.283725 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 17773
I20260927 21:31:38.301313 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 17907
Sep 27 21:31:38 dist-test-slave-bxm0 krb5kdc[17533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 21:31:38.318348 18049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:38.302197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42548 (local address 127.0.97.3:46553)
0927 21:31:38.302400 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:38.302409 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:38.302506 (+ 97us) server_negotiation.cc:407] Connection header received
0927 21:31:38.302725 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:38.302729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:38.302795 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:38.302906 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:38.304202 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:38.305070 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:38.306598 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:38.307059 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:38.307222 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:38.311691 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:38.311704 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:38.311723 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:38.311768 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:38.315062 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:38.315928 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:38.315936 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:38.315939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:38.316115 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:38.316546 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:38.316553 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:38.316561 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:38.316946 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:38.317291 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:38.317594 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 21:31:38.317976 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 21:31:38.319192 18030 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:38.442045 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 17907
I20260927 21:31:38.461472 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 17546
I20260927 21:31:38.462644 17608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:38.572448 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 17546
2026-09-27T21:31:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4546 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T21:31:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:31:38Z Disabled control of system clock
I20260927 21:31:38.651970 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33355
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:31:38.989003 18087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:38.989280 18087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:38.989432 18087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:38.996955 18087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:31:38.997094 18087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:38.997165 18087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:38.997268 18087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:31:38.997334 18087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:31:39.006932 18087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33355
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:39.008236 18087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:39.010062 18087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:39.021888 18093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:39.021971 18092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:39.022439 18095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:39.022989 18087 server_base.cc:1034] running on GCE node
I20260927 21:31:39.023818 18087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:39.025516 18087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:39.026768 18087 hybrid_clock.cc:648] HybridClock initialized: now 1790544699026728 us; error 35 us; skew 500 ppm
I20260927 21:31:39.027137 18087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:31:39.029312 18087 webserver.cc:460] Webserver started at http://127.0.97.62:36145/ using document root <none> and password file <none>
I20260927 21:31:39.030068 18087 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:39.030194 18087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:39.030522 18087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:31:39.033324 18087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "86793e9ea783479e9ec156499cfd349c"
format_stamp: "Formatted at 2026-09-27 21:31:39 on dist-test-slave-bxm0"
server_key: "17105e097b03e1dd17c9c66def2c5a84"
server_key_iv: "082ba2ba028bfe87aced7d7639340646"
server_key_version: "encryptionkey@0"
I20260927 21:31:39.034133 18087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "86793e9ea783479e9ec156499cfd349c"
format_stamp: "Formatted at 2026-09-27 21:31:39 on dist-test-slave-bxm0"
server_key: "17105e097b03e1dd17c9c66def2c5a84"
server_key_iv: "082ba2ba028bfe87aced7d7639340646"
server_key_version: "encryptionkey@0"
I20260927 21:31:39.039031 18087 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 21:31:39.042359 18101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:39.043702 18087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:31:39.043898 18087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "86793e9ea783479e9ec156499cfd349c"
format_stamp: "Formatted at 2026-09-27 21:31:39 on dist-test-slave-bxm0"
server_key: "17105e097b03e1dd17c9c66def2c5a84"
server_key_iv: "082ba2ba028bfe87aced7d7639340646"
server_key_version: "encryptionkey@0"
I20260927 21:31:39.044107 18087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:39.062418 18087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:39.063549 18087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:39.063907 18087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:39.082942 18087 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33355
I20260927 21:31:39.082952 18152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33355 every 8 connection(s)
I20260927 21:31:39.084467 18087 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:31:39.086401 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 18087
I20260927 21:31:39.086577 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:31:39.087163 388 external_mini_cluster.cc:1428] Setting key 3d3a74235129cbf73de3ec47c50670ae
I20260927 21:31:39.089481 18153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:31:39.103053 18156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:39.089167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57762 (local address 127.0.97.62:33355)
0927 21:31:39.090292 (+ 1125us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:39.090306 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:39.090356 (+ 50us) server_negotiation.cc:407] Connection header received
0927 21:31:39.092541 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:39.092570 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:39.093211 (+ 641us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:39.093677 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:39.094214 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:39.096250 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:39.097725 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:39.098733 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:39.098923 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:39.100146 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:39.100157 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:39.100180 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:31:39.100923 (+ 743us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:39.101022 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:31:39.101053 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:39.101181 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:39.101395 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 21:31:39.101776 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":396,"threads_started":1}
I20260927 21:31:39.104251 18153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86793e9ea783479e9ec156499cfd349c: Bootstrap starting.
I20260927 21:31:39.107535 18153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86793e9ea783479e9ec156499cfd349c: Neither blocks nor log segments found. Creating new log.
I20260927 21:31:39.108695 18153 log.cc:824] T 00000000000000000000000000000000 P 86793e9ea783479e9ec156499cfd349c: Log is configured to *not* fsync() on all Append() calls
I20260927 21:31:39.111407 18153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86793e9ea783479e9ec156499cfd349c: No bootstrap required, opened a new log
I20260927 21:31:39.118849 18153 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86793e9ea783479e9ec156499cfd349c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86793e9ea783479e9ec156499cfd349c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33355 } }
I20260927 21:31:39.119199 18153 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86793e9ea783479e9ec156499cfd349c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:31:39.119305 18153 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86793e9ea783479e9ec156499cfd349c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86793e9ea783479e9ec156499cfd349c, State: Initialized, Role: FOLLOWER
I20260927 21:31:39.119900 18153 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86793e9ea783479e9ec156499cfd349c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86793e9ea783479e9ec156499cfd349c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33355 } }
I20260927 21:31:39.120127 18153 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86793e9ea783479e9ec156499cfd349c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:31:39.120262 18153 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86793e9ea783479e9ec156499cfd349c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:31:39.120438 18153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86793e9ea783479e9ec156499cfd349c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:31:39.122920 18153 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86793e9ea783479e9ec156499cfd349c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86793e9ea783479e9ec156499cfd349c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33355 } }
I20260927 21:31:39.123409 18153 leader_election.cc:302] T 00000000000000000000000000000000 P 86793e9ea783479e9ec156499cfd349c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86793e9ea783479e9ec156499cfd349c; no voters:
I20260927 21:31:39.123960 18153 leader_election.cc:288] T 00000000000000000000000000000000 P 86793e9ea783479e9ec156499cfd349c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:31:39.124119 18158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86793e9ea783479e9ec156499cfd349c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:31:39.124832 18158 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86793e9ea783479e9ec156499cfd349c [term 1 LEADER]: Becoming Leader. State: Replica: 86793e9ea783479e9ec156499cfd349c, State: Running, Role: LEADER
I20260927 21:31:39.125329 18158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86793e9ea783479e9ec156499cfd349c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86793e9ea783479e9ec156499cfd349c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33355 } }
I20260927 21:31:39.126060 18153 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86793e9ea783479e9ec156499cfd349c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:31:39.128182 18159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86793e9ea783479e9ec156499cfd349c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86793e9ea783479e9ec156499cfd349c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86793e9ea783479e9ec156499cfd349c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33355 } } }
I20260927 21:31:39.128710 18159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86793e9ea783479e9ec156499cfd349c [sys.catalog]: This master's current role is: LEADER
I20260927 21:31:39.128207 18160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86793e9ea783479e9ec156499cfd349c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86793e9ea783479e9ec156499cfd349c. Latest consensus state: current_term: 1 leader_uuid: "86793e9ea783479e9ec156499cfd349c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86793e9ea783479e9ec156499cfd349c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33355 } } }
I20260927 21:31:39.128968 18160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86793e9ea783479e9ec156499cfd349c [sys.catalog]: This master's current role is: LEADER
I20260927 21:31:39.130223 18165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:31:39.135434 18165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:31:39.144374 18165 catalog_manager.cc:1347] Generated new cluster ID: 931da614ab27400eb0726ba59052bf9a
I20260927 21:31:39.144549 18165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:31:39.163057 18165 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:31:39.164420 18165 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:31:39.176327 18165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86793e9ea783479e9ec156499cfd349c: Generated new TSK 0
I20260927 21:31:39.177214 18165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:31:39.190831 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33355
--builtin_ntp_servers=127.0.97.20:45849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:31:39.547880 18177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:39.548177 18177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:39.548332 18177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:39.555842 18177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:39.555953 18177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:39.556128 18177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:31:39.565681 18177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:39.567050 18177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:39.568977 18177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:39.582624 18185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:39.582602 18183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:39.582571 18182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:39.587141 18177 server_base.cc:1034] running on GCE node
I20260927 21:31:39.587841 18177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:39.588883 18177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:39.590097 18177 hybrid_clock.cc:648] HybridClock initialized: now 1790544699590042 us; error 45 us; skew 500 ppm
I20260927 21:31:39.590433 18177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:31:39.592813 18177 webserver.cc:460] Webserver started at http://127.0.97.1:45353/ using document root <none> and password file <none>
I20260927 21:31:39.593497 18177 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:39.593583 18177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:39.593928 18177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:31:39.596840 18177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-0/data/instance:
uuid: "e2460764519f4242bf1b00a9c701098a"
format_stamp: "Formatted at 2026-09-27 21:31:39 on dist-test-slave-bxm0"
server_key: "01ada11b889ce5ff3b0cd05995602a78"
server_key_iv: "5ea2f96e24c2334352fb01ca280ce4fe"
server_key_version: "encryptionkey@0"
I20260927 21:31:39.597599 18177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e2460764519f4242bf1b00a9c701098a"
format_stamp: "Formatted at 2026-09-27 21:31:39 on dist-test-slave-bxm0"
server_key: "01ada11b889ce5ff3b0cd05995602a78"
server_key_iv: "5ea2f96e24c2334352fb01ca280ce4fe"
server_key_version: "encryptionkey@0"
I20260927 21:31:39.603091 18177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 21:31:39.607134 18191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:39.608517 18177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:31:39.608735 18177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "e2460764519f4242bf1b00a9c701098a"
format_stamp: "Formatted at 2026-09-27 21:31:39 on dist-test-slave-bxm0"
server_key: "01ada11b889ce5ff3b0cd05995602a78"
server_key_iv: "5ea2f96e24c2334352fb01ca280ce4fe"
server_key_version: "encryptionkey@0"
I20260927 21:31:39.608970 18177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:39.627715 18177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:39.628834 18177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:39.629122 18177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:39.634078 18177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:31:39.635807 18177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:31:39.635939 18177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:39.636081 18177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:31:39.636176 18177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:39.670198 18177 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39489
I20260927 21:31:39.670254 18303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39489 every 8 connection(s)
I20260927 21:31:39.671538 18177 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:31:39.679859 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 18177
I20260927 21:31:39.680058 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-0/wal/instance
I20260927 21:31:39.680646 388 external_mini_cluster.cc:1428] Setting key 2b878b31a2b6cfd51126fa73bf4a0052
I20260927 21:31:39.683907 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33355
--builtin_ntp_servers=127.0.97.20:45849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:31:39.685160 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:39.674329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36815 (local address 127.0.97.62:33355)
0927 21:31:39.674797 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:39.674802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:39.676110 (+ 1308us) server_negotiation.cc:407] Connection header received
0927 21:31:39.677916 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:39.677920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:39.677987 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:39.678088 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:39.679172 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:39.679861 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:39.682062 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:39.682627 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:39.682777 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:39.684367 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:39.684372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:39.684376 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:31:39.684395 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:39.684434 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:31:39.684447 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:39.684522 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:39.684793 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 21:31:39.684892 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":237,"threads_started":1}
I20260927 21:31:39.686794 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:39.674960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33355 (local address 127.0.97.1:36815)
0927 21:31:39.675948 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:39.675990 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:39.677598 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:39.678303 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:39.678312 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:39.678425 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:31:39.679016 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:39.679033 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:39.680015 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:39.680022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:39.681917 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:39.681929 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:39.682878 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:39.682887 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:39.683212 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:39.683983 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:31:39.683988 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:31:39.684049 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:31:39.684176 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:31:39.684553 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:31:39.684567 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:31:39.684579 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 21:31:39.684781 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 21:31:39.685279 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":333,"threads_started":1}
I20260927 21:31:39.688445 18304 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33355
I20260927 21:31:39.688774 18304 heartbeater.cc:464] Registering TS with master...
I20260927 21:31:39.689569 18304 heartbeater.cc:511] Master 127.0.97.62:33355 requested a full tablet report, sending...
I20260927 21:31:39.691715 18118 ts_manager.cc:194] Registered new tserver with Master: e2460764519f4242bf1b00a9c701098a (127.0.97.1:39489)
I20260927 21:31:39.693471 18118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:36815
W20260927 21:31:40.032938 18308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:40.033229 18308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:40.033314 18308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:40.040732 18308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:40.040844 18308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:40.040952 18308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:31:40.050616 18308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:40.051887 18308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:40.053722 18308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:40.067817 18313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:40.067814 18314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:40.067754 18316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:40.068902 18308 server_base.cc:1034] running on GCE node
I20260927 21:31:40.069505 18308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:40.070541 18308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:40.071761 18308 hybrid_clock.cc:648] HybridClock initialized: now 1790544700071715 us; error 33 us; skew 500 ppm
I20260927 21:31:40.072110 18308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:31:40.074481 18308 webserver.cc:460] Webserver started at http://127.0.97.2:44395/ using document root <none> and password file <none>
I20260927 21:31:40.075178 18308 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:40.075296 18308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:40.075632 18308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:31:40.078588 18308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-1/data/instance:
uuid: "4725bc4b08174e5faae20f92797fcc6f"
format_stamp: "Formatted at 2026-09-27 21:31:40 on dist-test-slave-bxm0"
server_key: "7d855beda9f2ca5e691170e525b4f8c2"
server_key_iv: "6236add4a9c479761f13ffd3b37579f9"
server_key_version: "encryptionkey@0"
I20260927 21:31:40.079403 18308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4725bc4b08174e5faae20f92797fcc6f"
format_stamp: "Formatted at 2026-09-27 21:31:40 on dist-test-slave-bxm0"
server_key: "7d855beda9f2ca5e691170e525b4f8c2"
server_key_iv: "6236add4a9c479761f13ffd3b37579f9"
server_key_version: "encryptionkey@0"
I20260927 21:31:40.084988 18308 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:31:40.088838 18322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:40.090917 18308 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260927 21:31:40.091110 18308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "4725bc4b08174e5faae20f92797fcc6f"
format_stamp: "Formatted at 2026-09-27 21:31:40 on dist-test-slave-bxm0"
server_key: "7d855beda9f2ca5e691170e525b4f8c2"
server_key_iv: "6236add4a9c479761f13ffd3b37579f9"
server_key_version: "encryptionkey@0"
I20260927 21:31:40.091383 18308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:40.124861 18308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:40.125748 18308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:40.125996 18308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:40.127038 18308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:31:40.128561 18308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:31:40.128645 18308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:40.128722 18308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:31:40.128779 18308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:40.163431 18308 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42011
I20260927 21:31:40.163487 18434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42011 every 8 connection(s)
I20260927 21:31:40.164904 18308 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:31:40.172207 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 18308
I20260927 21:31:40.172408 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-1/wal/instance
I20260927 21:31:40.172945 388 external_mini_cluster.cc:1428] Setting key 57af71c783d8e074433b5acf0f9ed2e8
I20260927 21:31:40.176012 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33355
--builtin_ntp_servers=127.0.97.20:45849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:31:40.179092 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:40.167552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35333 (local address 127.0.97.62:33355)
0927 21:31:40.167811 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:40.167819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:40.169305 (+ 1486us) server_negotiation.cc:407] Connection header received
0927 21:31:40.171043 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:40.171047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:40.171116 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:40.171213 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:40.172268 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:40.173386 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:40.175537 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:40.176198 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:40.176362 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:40.178343 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:40.178352 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:40.178355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:31:40.178375 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:40.178409 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:31:40.178421 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:40.178495 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:40.178787 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 21:31:40.178868 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 21:31:40.180961 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:40.168213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33355 (local address 127.0.97.2:35333)
0927 21:31:40.169129 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:40.169172 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:40.170786 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:40.171410 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:40.171419 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:40.171502 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:31:40.172084 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:40.172100 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:40.173565 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:40.173571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:40.175395 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:40.175407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:40.176670 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:40.176678 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:40.177050 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:40.177915 (+ 865us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:31:40.177919 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:31:40.177999 (+ 80us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:31:40.178132 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:31:40.178524 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:31:40.178537 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:31:40.178547 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 21:31:40.178765 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 21:31:40.179247 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":316,"threads_started":1}
I20260927 21:31:40.182468 18435 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33355
I20260927 21:31:40.182807 18435 heartbeater.cc:464] Registering TS with master...
I20260927 21:31:40.183553 18435 heartbeater.cc:511] Master 127.0.97.62:33355 requested a full tablet report, sending...
I20260927 21:31:40.184964 18118 ts_manager.cc:194] Registered new tserver with Master: 4725bc4b08174e5faae20f92797fcc6f (127.0.97.2:42011)
I20260927 21:31:40.185847 18118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:35333
W20260927 21:31:40.511333 18438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:40.511631 18438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:40.511720 18438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:40.519409 18438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:40.519521 18438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:40.519632 18438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:31:40.529107 18438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:40.530390 18438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:40.532308 18438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:40.545636 18443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:40.545576 18444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:40.545573 18446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:40.546463 18438 server_base.cc:1034] running on GCE node
I20260927 21:31:40.547133 18438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:40.548306 18438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:40.549538 18438 hybrid_clock.cc:648] HybridClock initialized: now 1790544700549478 us; error 55 us; skew 500 ppm
I20260927 21:31:40.549888 18438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:31:40.552083 18438 webserver.cc:460] Webserver started at http://127.0.97.3:46785/ using document root <none> and password file <none>
I20260927 21:31:40.552791 18438 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:40.552917 18438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:40.553259 18438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:31:40.556137 18438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-2/data/instance:
uuid: "fda19c636e434e8982575f62c58fcc88"
format_stamp: "Formatted at 2026-09-27 21:31:40 on dist-test-slave-bxm0"
server_key: "f47a0060574dcd51fc52a9c314d54099"
server_key_iv: "6e22501dda701ca511f9e315505f51f9"
server_key_version: "encryptionkey@0"
I20260927 21:31:40.556917 18438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fda19c636e434e8982575f62c58fcc88"
format_stamp: "Formatted at 2026-09-27 21:31:40 on dist-test-slave-bxm0"
server_key: "f47a0060574dcd51fc52a9c314d54099"
server_key_iv: "6e22501dda701ca511f9e315505f51f9"
server_key_version: "encryptionkey@0"
I20260927 21:31:40.562026 18438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 21:31:40.565471 18452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:40.566871 18438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:31:40.567072 18438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "fda19c636e434e8982575f62c58fcc88"
format_stamp: "Formatted at 2026-09-27 21:31:40 on dist-test-slave-bxm0"
server_key: "f47a0060574dcd51fc52a9c314d54099"
server_key_iv: "6e22501dda701ca511f9e315505f51f9"
server_key_version: "encryptionkey@0"
I20260927 21:31:40.567314 18438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:40.587419 18438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:40.588270 18438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:40.588552 18438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:40.589504 18438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:31:40.591027 18438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:31:40.591147 18438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:40.591274 18438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:31:40.591360 18438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:40.625464 18438 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41045
I20260927 21:31:40.625550 18564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41045 every 8 connection(s)
I20260927 21:31:40.626899 18438 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:31:40.636143 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 18438
I20260927 21:31:40.636335 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-2/wal/instance
I20260927 21:31:40.636765 388 external_mini_cluster.cc:1428] Setting key de502a4a7d67e77bd67883e93eff6ab3
I20260927 21:31:40.640285 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:40.629716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44909 (local address 127.0.97.62:33355)
0927 21:31:40.629896 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:40.629901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:40.631372 (+ 1471us) server_negotiation.cc:407] Connection header received
0927 21:31:40.633286 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:40.633292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:40.633392 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:40.633513 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:40.634639 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:40.635375 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:40.637081 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:40.637702 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:40.637850 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:40.639479 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:40.639485 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:40.639490 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:31:40.639522 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:40.639565 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:31:40.639583 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:40.639709 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:40.639951 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 21:31:40.640042 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:31:40.641356 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:40.630137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33355 (local address 127.0.97.3:44909)
0927 21:31:40.631210 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:40.631247 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:40.633001 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:40.633783 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:40.633792 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:40.633882 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:31:40.634464 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:40.634481 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:40.635548 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:40.635551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:40.636942 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:40.636956 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:40.637963 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:40.637972 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:40.638319 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:40.639166 (+ 847us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:31:40.639169 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:31:40.639208 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:31:40.639292 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:31:40.639723 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:31:40.639731 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:31:40.639737 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 21:31:40.639930 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 21:31:40.640268 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":312,"threads_started":1}
I20260927 21:31:40.642800 18565 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33355
I20260927 21:31:40.643105 18565 heartbeater.cc:464] Registering TS with master...
I20260927 21:31:40.643826 18565 heartbeater.cc:511] Master 127.0.97.62:33355 requested a full tablet report, sending...
I20260927 21:31:40.645231 18118 ts_manager.cc:194] Registered new tserver with Master: fda19c636e434e8982575f62c58fcc88 (127.0.97.3:41045)
I20260927 21:31:40.646075 18118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:44909
I20260927 21:31:40.652169 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:31:40.661871 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:40.656220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57772 (local address 127.0.97.62:33355)
0927 21:31:40.656475 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:40.656481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:40.656760 (+ 279us) server_negotiation.cc:407] Connection header received
0927 21:31:40.656898 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:40.656902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:40.656961 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:40.657039 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:40.657478 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:40.658216 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:40.659614 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:40.660029 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:40.660119 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:40.661133 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:40.661138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:40.661141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:31:40.661161 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:40.661202 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:31:40.661214 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:40.661281 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:40.661432 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 21:31:40.661506 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 21:31:40.665978 18118 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:31:40.674971 18118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:31:40.697667 18304 heartbeater.cc:503] Master 127.0.97.62:33355 was elected leader, sending a full tablet report...
I20260927 21:31:40.714959 18579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:40.687900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35932 (local address 127.0.97.3:41045)
0927 21:31:40.705553 (+ 17653us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:40.705564 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:40.705622 (+ 58us) server_negotiation.cc:407] Connection header received
0927 21:31:40.705749 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:40.705759 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:40.706054 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:40.707247 (+ 1193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:40.707607 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:40.709255 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:40.711974 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:40.713563 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:40.713813 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:40.714104 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:40.714397 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 21:31:40.714584 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17463,"thread_start_us":17306,"threads_started":1}
I20260927 21:31:40.715065 18578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:40.687586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41045 (local address 127.0.0.1:35932)
0927 21:31:40.699041 (+ 11455us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:40.699119 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:40.699380 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:40.706350 (+ 6970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:40.706361 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:40.706417 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:40.706968 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:40.706986 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:40.709419 (+ 2433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:40.709425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:40.711756 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:40.711771 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:40.713881 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:40.713890 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:40.714117 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:40.714183 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 21:31:40.714334 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 21:31:40.714565 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11044,"mutex_wait_us":2,"thread_start_us":18040,"threads_started":1}
I20260927 21:31:40.718413 18581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:40.689114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42011 (local address 127.0.0.1:50784)
0927 21:31:40.708619 (+ 19505us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:40.708645 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:40.708809 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:40.709860 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:40.709863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:40.709883 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:40.710286 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:40.710300 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:40.712121 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:40.712126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:40.715006 (+ 2880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:40.715023 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:40.717399 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:40.717405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:40.717669 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:40.717710 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 21:31:40.717791 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 21:31:40.717954 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19325,"mutex_wait_us":35,"thread_start_us":19123,"threads_started":1}
I20260927 21:31:40.718745 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:40.687937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39489 (local address 127.0.0.1:36032)
0927 21:31:40.705899 (+ 17962us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:40.705922 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:40.706046 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:40.710335 (+ 4289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:40.710345 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:40.710373 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:40.710875 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:40.710888 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:40.712798 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:40.712803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:40.715092 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:40.715113 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:40.717834 (+ 2721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:40.717841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:40.718028 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:40.718063 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 21:31:40.718157 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 21:31:40.718298 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17833,"mutex_wait_us":81,"thread_start_us":10475,"threads_started":1}
I20260927 21:31:40.718745 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:40.688634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36032 (local address 127.0.97.1:39489)
0927 21:31:40.709767 (+ 21133us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:40.709776 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:40.709804 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:31:40.709891 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:40.709899 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:40.710118 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:40.710314 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:40.711060 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:40.712628 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:40.715323 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:40.717597 (+ 2274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:40.717765 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:40.718016 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:40.718218 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 21:31:40.718392 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20907,"thread_start_us":20753,"threads_started":1}
I20260927 21:31:40.719267 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:40.689386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50784 (local address 127.0.97.2:42011)
0927 21:31:40.708048 (+ 18662us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:40.708061 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:40.708991 (+ 930us) server_negotiation.cc:407] Connection header received
0927 21:31:40.709108 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:40.709121 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:40.709438 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:40.709655 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:40.710539 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:40.711881 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:40.715558 (+ 3677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:40.716843 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:40.716987 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:40.717251 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:40.718731 (+ 1480us) server_negotiation.cc:299] Negotiation successful
0927 21:31:40.718921 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18458,"thread_start_us":18353,"threads_started":1}
I20260927 21:31:40.719448 18500 tablet_service.cc:1467] Processing CreateTablet for tablet eed44d29976946ae995738b2933059d1 (DEFAULT_TABLE table=test-table [id=3829a3fe2629442197ecb47b4df70933]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:31:40.721055 18500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eed44d29976946ae995738b2933059d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:31:40.722504 18239 tablet_service.cc:1467] Processing CreateTablet for tablet eed44d29976946ae995738b2933059d1 (DEFAULT_TABLE table=test-table [id=3829a3fe2629442197ecb47b4df70933]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:31:40.722787 18370 tablet_service.cc:1467] Processing CreateTablet for tablet eed44d29976946ae995738b2933059d1 (DEFAULT_TABLE table=test-table [id=3829a3fe2629442197ecb47b4df70933]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:31:40.723943 18370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eed44d29976946ae995738b2933059d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:31:40.724108 18239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eed44d29976946ae995738b2933059d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:31:40.736541 18583 tablet_bootstrap.cc:492] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f: Bootstrap starting.
I20260927 21:31:40.737602 18584 tablet_bootstrap.cc:492] T eed44d29976946ae995738b2933059d1 P fda19c636e434e8982575f62c58fcc88: Bootstrap starting.
I20260927 21:31:40.740988 18585 tablet_bootstrap.cc:492] T eed44d29976946ae995738b2933059d1 P e2460764519f4242bf1b00a9c701098a: Bootstrap starting.
I20260927 21:31:40.741673 18584 tablet_bootstrap.cc:654] T eed44d29976946ae995738b2933059d1 P fda19c636e434e8982575f62c58fcc88: Neither blocks nor log segments found. Creating new log.
I20260927 21:31:40.741712 18583 tablet_bootstrap.cc:654] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f: Neither blocks nor log segments found. Creating new log.
I20260927 21:31:40.743091 18584 log.cc:824] T eed44d29976946ae995738b2933059d1 P fda19c636e434e8982575f62c58fcc88: Log is configured to *not* fsync() on all Append() calls
I20260927 21:31:40.743091 18583 log.cc:824] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f: Log is configured to *not* fsync() on all Append() calls
I20260927 21:31:40.744554 18585 tablet_bootstrap.cc:654] T eed44d29976946ae995738b2933059d1 P e2460764519f4242bf1b00a9c701098a: Neither blocks nor log segments found. Creating new log.
I20260927 21:31:40.745677 18585 log.cc:824] T eed44d29976946ae995738b2933059d1 P e2460764519f4242bf1b00a9c701098a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:31:40.748745 18584 tablet_bootstrap.cc:492] T eed44d29976946ae995738b2933059d1 P fda19c636e434e8982575f62c58fcc88: No bootstrap required, opened a new log
I20260927 21:31:40.748821 18583 tablet_bootstrap.cc:492] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f: No bootstrap required, opened a new log
I20260927 21:31:40.749055 18584 ts_tablet_manager.cc:1397] T eed44d29976946ae995738b2933059d1 P fda19c636e434e8982575f62c58fcc88: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260927 21:31:40.749117 18583 ts_tablet_manager.cc:1397] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20260927 21:31:40.750540 18585 tablet_bootstrap.cc:492] T eed44d29976946ae995738b2933059d1 P e2460764519f4242bf1b00a9c701098a: No bootstrap required, opened a new log
I20260927 21:31:40.750828 18585 ts_tablet_manager.cc:1397] T eed44d29976946ae995738b2933059d1 P e2460764519f4242bf1b00a9c701098a: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 21:31:40.757893 18584 raft_consensus.cc:348] T eed44d29976946ae995738b2933059d1 P fda19c636e434e8982575f62c58fcc88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda19c636e434e8982575f62c58fcc88" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41045 } } peers { permanent_uuid: "e2460764519f4242bf1b00a9c701098a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39489 } } peers { permanent_uuid: "4725bc4b08174e5faae20f92797fcc6f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } }
I20260927 21:31:40.758311 18584 raft_consensus.cc:374] T eed44d29976946ae995738b2933059d1 P fda19c636e434e8982575f62c58fcc88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:31:40.758426 18584 raft_consensus.cc:729] T eed44d29976946ae995738b2933059d1 P fda19c636e434e8982575f62c58fcc88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fda19c636e434e8982575f62c58fcc88, State: Initialized, Role: FOLLOWER
I20260927 21:31:40.759119 18584 consensus_queue.cc:260] T eed44d29976946ae995738b2933059d1 P fda19c636e434e8982575f62c58fcc88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda19c636e434e8982575f62c58fcc88" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41045 } } peers { permanent_uuid: "e2460764519f4242bf1b00a9c701098a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39489 } } peers { permanent_uuid: "4725bc4b08174e5faae20f92797fcc6f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } }
I20260927 21:31:40.760316 18565 heartbeater.cc:503] Master 127.0.97.62:33355 was elected leader, sending a full tablet report...
I20260927 21:31:40.761104 18584 ts_tablet_manager.cc:1428] T eed44d29976946ae995738b2933059d1 P fda19c636e434e8982575f62c58fcc88: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260927 21:31:40.760819 18583 raft_consensus.cc:348] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda19c636e434e8982575f62c58fcc88" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41045 } } peers { permanent_uuid: "e2460764519f4242bf1b00a9c701098a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39489 } } peers { permanent_uuid: "4725bc4b08174e5faae20f92797fcc6f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } }
I20260927 21:31:40.761305 18583 raft_consensus.cc:374] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:31:40.761408 18583 raft_consensus.cc:729] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4725bc4b08174e5faae20f92797fcc6f, State: Initialized, Role: FOLLOWER
I20260927 21:31:40.762094 18583 consensus_queue.cc:260] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda19c636e434e8982575f62c58fcc88" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41045 } } peers { permanent_uuid: "e2460764519f4242bf1b00a9c701098a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39489 } } peers { permanent_uuid: "4725bc4b08174e5faae20f92797fcc6f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } }
I20260927 21:31:40.762465 18585 raft_consensus.cc:348] T eed44d29976946ae995738b2933059d1 P e2460764519f4242bf1b00a9c701098a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda19c636e434e8982575f62c58fcc88" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41045 } } peers { permanent_uuid: "e2460764519f4242bf1b00a9c701098a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39489 } } peers { permanent_uuid: "4725bc4b08174e5faae20f92797fcc6f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } }
I20260927 21:31:40.762965 18585 raft_consensus.cc:374] T eed44d29976946ae995738b2933059d1 P e2460764519f4242bf1b00a9c701098a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:31:40.763075 18585 raft_consensus.cc:729] T eed44d29976946ae995738b2933059d1 P e2460764519f4242bf1b00a9c701098a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2460764519f4242bf1b00a9c701098a, State: Initialized, Role: FOLLOWER
I20260927 21:31:40.763481 18435 heartbeater.cc:503] Master 127.0.97.62:33355 was elected leader, sending a full tablet report...
I20260927 21:31:40.764032 18583 ts_tablet_manager.cc:1428] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
I20260927 21:31:40.764629 18585 consensus_queue.cc:260] T eed44d29976946ae995738b2933059d1 P e2460764519f4242bf1b00a9c701098a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda19c636e434e8982575f62c58fcc88" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41045 } } peers { permanent_uuid: "e2460764519f4242bf1b00a9c701098a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39489 } } peers { permanent_uuid: "4725bc4b08174e5faae20f92797fcc6f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } }
I20260927 21:31:40.766589 18585 ts_tablet_manager.cc:1428] T eed44d29976946ae995738b2933059d1 P e2460764519f4242bf1b00a9c701098a: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
W20260927 21:31:40.878677 18566 tablet.cc:2369] T eed44d29976946ae995738b2933059d1 P fda19c636e434e8982575f62c58fcc88: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:31:40.917162 18436 tablet.cc:2369] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:31:40.924096 18305 tablet.cc:2369] T eed44d29976946ae995738b2933059d1 P e2460764519f4242bf1b00a9c701098a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:31:40.998224 18590 raft_consensus.cc:482] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:31:40.998588 18590 raft_consensus.cc:504] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda19c636e434e8982575f62c58fcc88" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41045 } } peers { permanent_uuid: "e2460764519f4242bf1b00a9c701098a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39489 } } peers { permanent_uuid: "4725bc4b08174e5faae20f92797fcc6f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } }
I20260927 21:31:41.000437 18590 leader_election.cc:288] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fda19c636e434e8982575f62c58fcc88 (127.0.97.3:41045), e2460764519f4242bf1b00a9c701098a (127.0.97.1:39489)
I20260927 21:31:41.007489 18579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:41.000999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57209 (local address 127.0.97.3:41045)
0927 21:31:41.001276 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:41.001285 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:41.001810 (+ 525us) server_negotiation.cc:407] Connection header received
0927 21:31:41.002042 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:41.002047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:41.002111 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:41.002213 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:41.002836 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:41.003589 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:41.005688 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:41.006473 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:41.006640 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:41.006755 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:41.007081 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 21:31:41.007222 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 21:31:41.007489 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:41.000826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40037 (local address 127.0.97.1:39489)
0927 21:31:41.001072 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:41.001101 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:41.001969 (+ 868us) server_negotiation.cc:407] Connection header received
0927 21:31:41.002189 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:41.002193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:41.002264 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:41.002366 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:41.003025 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:41.003707 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:41.005674 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:41.006442 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:41.006611 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:41.006717 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:41.007082 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 21:31:41.007224 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 21:31:41.007648 18592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:41.000696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41045 (local address 127.0.97.2:57209)
0927 21:31:41.001686 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:41.001712 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:41.001887 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:41.002231 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:41.002235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:41.002258 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:41.002681 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:41.002690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:41.003728 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:41.003735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:41.005452 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:41.005465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:41.006675 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:41.006682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:41.006904 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:41.006941 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 21:31:41.007043 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 21:31:41.007239 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":135,"spinlock_wait_cycles":19200,"thread_start_us":611,"threads_started":1}
I20260927 21:31:41.007820 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:41.000900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39489 (local address 127.0.97.2:40037)
0927 21:31:41.001857 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:41.001872 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:41.002035 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:41.002383 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:41.002387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:41.002426 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:41.002874 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:41.002883 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:41.003850 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:41.003856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:41.005438 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:41.005451 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:41.006661 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:41.006668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:41.006906 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:41.006947 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 21:31:41.007058 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 21:31:41.007225 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":166,"thread_start_us":619,"threads_started":1}
I20260927 21:31:41.008783 18520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed44d29976946ae995738b2933059d1" candidate_uuid: "4725bc4b08174e5faae20f92797fcc6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fda19c636e434e8982575f62c58fcc88" is_pre_election: true
I20260927 21:31:41.008818 18259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed44d29976946ae995738b2933059d1" candidate_uuid: "4725bc4b08174e5faae20f92797fcc6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2460764519f4242bf1b00a9c701098a" is_pre_election: true
I20260927 21:31:41.009305 18259 raft_consensus.cc:2454] T eed44d29976946ae995738b2933059d1 P e2460764519f4242bf1b00a9c701098a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4725bc4b08174e5faae20f92797fcc6f in term 0.
I20260927 21:31:41.009306 18520 raft_consensus.cc:2454] T eed44d29976946ae995738b2933059d1 P fda19c636e434e8982575f62c58fcc88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4725bc4b08174e5faae20f92797fcc6f in term 0.
I20260927 21:31:41.010051 18324 leader_election.cc:302] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4725bc4b08174e5faae20f92797fcc6f, e2460764519f4242bf1b00a9c701098a, fda19c636e434e8982575f62c58fcc88; no voters:
I20260927 21:31:41.010475 18590 raft_consensus.cc:2781] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:31:41.010632 18590 raft_consensus.cc:482] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:31:41.010751 18590 raft_consensus.cc:3037] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:31:41.013116 18590 raft_consensus.cc:504] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda19c636e434e8982575f62c58fcc88" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41045 } } peers { permanent_uuid: "e2460764519f4242bf1b00a9c701098a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39489 } } peers { permanent_uuid: "4725bc4b08174e5faae20f92797fcc6f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } }
I20260927 21:31:41.013850 18590 leader_election.cc:288] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f [CANDIDATE]: Term 1 election: Requested vote from peers fda19c636e434e8982575f62c58fcc88 (127.0.97.3:41045), e2460764519f4242bf1b00a9c701098a (127.0.97.1:39489)
I20260927 21:31:41.014412 18520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed44d29976946ae995738b2933059d1" candidate_uuid: "4725bc4b08174e5faae20f92797fcc6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fda19c636e434e8982575f62c58fcc88"
I20260927 21:31:41.014441 18259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed44d29976946ae995738b2933059d1" candidate_uuid: "4725bc4b08174e5faae20f92797fcc6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2460764519f4242bf1b00a9c701098a"
I20260927 21:31:41.014621 18520 raft_consensus.cc:3037] T eed44d29976946ae995738b2933059d1 P fda19c636e434e8982575f62c58fcc88 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:31:41.014691 18259 raft_consensus.cc:3037] T eed44d29976946ae995738b2933059d1 P e2460764519f4242bf1b00a9c701098a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:31:41.017100 18520 raft_consensus.cc:2454] T eed44d29976946ae995738b2933059d1 P fda19c636e434e8982575f62c58fcc88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4725bc4b08174e5faae20f92797fcc6f in term 1.
I20260927 21:31:41.017104 18259 raft_consensus.cc:2454] T eed44d29976946ae995738b2933059d1 P e2460764519f4242bf1b00a9c701098a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4725bc4b08174e5faae20f92797fcc6f in term 1.
I20260927 21:31:41.017688 18323 leader_election.cc:302] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4725bc4b08174e5faae20f92797fcc6f, e2460764519f4242bf1b00a9c701098a; no voters:
I20260927 21:31:41.018100 18590 raft_consensus.cc:2781] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:31:41.018579 18590 raft_consensus.cc:686] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f [term 1 LEADER]: Becoming Leader. State: Replica: 4725bc4b08174e5faae20f92797fcc6f, State: Running, Role: LEADER
I20260927 21:31:41.019402 18590 consensus_queue.cc:237] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda19c636e434e8982575f62c58fcc88" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41045 } } peers { permanent_uuid: "e2460764519f4242bf1b00a9c701098a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39489 } } peers { permanent_uuid: "4725bc4b08174e5faae20f92797fcc6f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } }
I20260927 21:31:41.024171 18115 catalog_manager.cc:5495] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4725bc4b08174e5faae20f92797fcc6f (127.0.97.2). New cstate: current_term: 1 leader_uuid: "4725bc4b08174e5faae20f92797fcc6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda19c636e434e8982575f62c58fcc88" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2460764519f4242bf1b00a9c701098a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4725bc4b08174e5faae20f92797fcc6f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } health_report { overall_health: HEALTHY } } }
I20260927 21:31:41.083935 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:41.077450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50794 (local address 127.0.97.2:42011)
0927 21:31:41.077750 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:41.077758 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:41.077886 (+ 128us) server_negotiation.cc:407] Connection header received
0927 21:31:41.078192 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:41.078198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:41.078297 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:41.078477 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:31:41.078903 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:41.079709 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:41.081363 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:41.081913 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:41.082066 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:41.082774 (+ 708us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:31:41.083307 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:31:41.083425 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:41.083572 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 21:31:41.083665 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 21:31:41.093539 18520 raft_consensus.cc:1261] T eed44d29976946ae995738b2933059d1 P fda19c636e434e8982575f62c58fcc88 [term 1 FOLLOWER]: Refusing update from remote peer 4725bc4b08174e5faae20f92797fcc6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:31:41.093539 18259 raft_consensus.cc:1261] T eed44d29976946ae995738b2933059d1 P e2460764519f4242bf1b00a9c701098a [term 1 FOLLOWER]: Refusing update from remote peer 4725bc4b08174e5faae20f92797fcc6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:31:41.094795 18590 consensus_queue.cc:1035] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fda19c636e434e8982575f62c58fcc88" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:31:41.095634 18594 consensus_queue.cc:1035] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2460764519f4242bf1b00a9c701098a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:31:41.121526 18115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:31:41.121932 18115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:31:41.126410 18115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 86793e9ea783479e9ec156499cfd349c: Sending DeleteTablet for 3 replicas of tablet eed44d29976946ae995738b2933059d1
I20260927 21:31:41.127620 18239 tablet_service.cc:1514] Processing DeleteTablet for tablet eed44d29976946ae995738b2933059d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:31:41 UTC) from {username='slave'} at 127.0.0.1:36032
I20260927 21:31:41.127933 18370 tablet_service.cc:1514] Processing DeleteTablet for tablet eed44d29976946ae995738b2933059d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:31:41 UTC) from {username='slave'} at 127.0.0.1:50784
I20260927 21:31:41.128393 18500 tablet_service.cc:1514] Processing DeleteTablet for tablet eed44d29976946ae995738b2933059d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:31:41 UTC) from {username='slave'} at 127.0.0.1:35932
I20260927 21:31:41.128966 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 18177
I20260927 21:31:41.154861 18609 tablet_replica.cc:330] T eed44d29976946ae995738b2933059d1 P fda19c636e434e8982575f62c58fcc88: stopping tablet replica
I20260927 21:31:41.155525 18609 raft_consensus.cc:2229] T eed44d29976946ae995738b2933059d1 P fda19c636e434e8982575f62c58fcc88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:31:41.156185 18609 raft_consensus.cc:2258] T eed44d29976946ae995738b2933059d1 P fda19c636e434e8982575f62c58fcc88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:31:41.158771 18609 ts_tablet_manager.cc:1905] T eed44d29976946ae995738b2933059d1 P fda19c636e434e8982575f62c58fcc88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:31:41.160823 18607 tablet_replica.cc:330] T eed44d29976946ae995738b2933059d1 P e2460764519f4242bf1b00a9c701098a: stopping tablet replica
I20260927 21:31:41.161512 18607 raft_consensus.cc:2229] T eed44d29976946ae995738b2933059d1 P e2460764519f4242bf1b00a9c701098a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:31:41.162213 18607 raft_consensus.cc:2258] T eed44d29976946ae995738b2933059d1 P e2460764519f4242bf1b00a9c701098a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:31:41.170758 18607 ts_tablet_manager.cc:1905] T eed44d29976946ae995738b2933059d1 P e2460764519f4242bf1b00a9c701098a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:31:41.172279 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:41.130876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36044 (local address 127.0.97.1:39489)
0927 21:31:41.131075 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:41.131081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:41.152650 (+ 21569us) server_negotiation.cc:407] Connection header received
0927 21:31:41.152865 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:41.152871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:41.153004 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:41.153148 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:41.153945 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:41.155159 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:41.157944 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:41.158617 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:41.158728 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:41.170229 (+ 11501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:41.170243 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:41.170260 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:31:41.171101 (+ 841us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:41.171170 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:31:41.171202 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:41.171311 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:41.171757 (+ 446us) server_negotiation.cc:299] Negotiation successful
0927 21:31:41.171894 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 21:31:41.174018 18299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:41.178063 18608 tablet_replica.cc:330] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f: stopping tablet replica
I20260927 21:31:41.178639 18608 raft_consensus.cc:2229] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:31:41.179494 18608 raft_consensus.cc:2258] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:31:41.182245 18609 ts_tablet_manager.cc:1918] T eed44d29976946ae995738b2933059d1 P fda19c636e434e8982575f62c58fcc88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:31:41.182472 18609 log.cc:1196] T eed44d29976946ae995738b2933059d1 P fda19c636e434e8982575f62c58fcc88: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-2/wal/wals/eed44d29976946ae995738b2933059d1
I20260927 21:31:41.183167 18609 ts_tablet_manager.cc:1939] T eed44d29976946ae995738b2933059d1 P fda19c636e434e8982575f62c58fcc88: Deleting consensus metadata
I20260927 21:31:41.185334 18608 ts_tablet_manager.cc:1905] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:31:41.185877 18103 catalog_manager.cc:4841] TS fda19c636e434e8982575f62c58fcc88 (127.0.97.3:41045): tablet eed44d29976946ae995738b2933059d1 (table test-table [id=3829a3fe2629442197ecb47b4df70933]) successfully deleted
I20260927 21:31:41.194168 18608 ts_tablet_manager.cc:1918] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:31:41.194487 18608 log.cc:1196] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790544390748963-388-0/minicluster-data/ts-1/wal/wals/eed44d29976946ae995738b2933059d1
I20260927 21:31:41.195300 18608 ts_tablet_manager.cc:1939] T eed44d29976946ae995738b2933059d1 P 4725bc4b08174e5faae20f92797fcc6f: Deleting consensus metadata
I20260927 21:31:41.197475 18105 catalog_manager.cc:4841] TS 4725bc4b08174e5faae20f92797fcc6f (127.0.97.2:42011): tablet eed44d29976946ae995738b2933059d1 (table test-table [id=3829a3fe2629442197ecb47b4df70933]) successfully deleted
I20260927 21:31:41.286978 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 18177
I20260927 21:31:41.311214 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 18308
W20260927 21:31:41.311169 18102 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39489 (error 108)
I20260927 21:31:41.313396 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:41.312679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39489 (local address 127.0.0.1:33708)
0927 21:31:41.313127 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:41.313232 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":232,"threads_started":1}
W20260927 21:31:41.313833 18102 catalog_manager.cc:4584] TS e2460764519f4242bf1b00a9c701098a (127.0.97.1:39489): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eed44d29976946ae995738b2933059d1: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39489: connect: Connection refused (error 111)
I20260927 21:31:41.319576 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:41.312393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49756 (local address 127.0.97.2:42011)
0927 21:31:41.312610 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:41.312617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:41.312636 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:31:41.312703 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:41.312709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:41.312815 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:41.312919 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:41.313548 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:41.314512 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:41.316403 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:41.316843 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:41.316943 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:41.318136 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:41.318145 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:41.318163 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:31:41.318793 (+ 630us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:41.318839 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:31:41.318858 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:41.318940 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:41.319109 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 21:31:41.319187 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 21:31:41.320327 18430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:41.367655 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:41.367172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39489 (local address 127.0.0.1:33720)
0927 21:31:41.367411 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:41.367504 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 21:31:41.368829 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:41.368414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39489 (local address 127.0.0.1:33736)
0927 21:31:41.368613 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:41.368696 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 21:31:41.429227 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:41.428760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39489 (local address 127.0.0.1:33748)
0927 21:31:41.428984 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:41.429072 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 21:31:41.430359 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:41.429992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39489 (local address 127.0.0.1:33750)
0927 21:31:41.430144 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:41.430223 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 21:31:41.434479 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 18308
I20260927 21:31:41.454667 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 18438
I20260927 21:31:41.461761 18579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:41.455437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.0.97.3:41045)
0927 21:31:41.455643 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:41.455651 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:41.455673 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:31:41.455885 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:41.455894 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:41.455976 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:41.456105 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:41.456503 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:41.457213 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:41.458617 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:41.459013 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:41.459146 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:41.460290 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:41.460300 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:41.460315 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:31:41.460891 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:41.460939 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:31:41.460959 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:41.461089 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:41.461234 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 21:31:41.461345 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 21:31:41.462399 18560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:41.510892 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:41.510399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39489 (local address 127.0.0.1:33752)
0927 21:31:41.510636 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:41.510722 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260927 21:31:41.512023 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:41.511582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39489 (local address 127.0.0.1:33760)
0927 21:31:41.511809 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:41.511889 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 21:31:41.551410 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 18438
I20260927 21:31:41.567886 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 18087
I20260927 21:31:41.568979 18148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:41.683463 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 18087
2026-09-27T21:31:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3109 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:31:41 dist-test-slave-bxm0 krb5kdc[18619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:31:41 dist-test-slave-bxm0 krb5kdc[18619](info): set up 2 sockets
Sep 27 21:31:41 dist-test-slave-bxm0 krb5kdc[18619](info): commencing operation
krb5kdc: starting...
W20260927 21:31:42.794137 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:31:42 dist-test-slave-bxm0 krb5kdc[18619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:31:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:31:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:31:43.048516 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45027
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45027
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:31:43.387054 18632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:43.387415 18632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:43.387590 18632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:43.395095 18632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:31:43.395248 18632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:43.395366 18632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:43.395471 18632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:31:43.395558 18632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:31:43.405678 18632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:43.407080 18632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:43.408448 18632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:43.421180 18637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:43.421270 18638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:43.421211 18640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:43.427616 18632 server_base.cc:1034] running on GCE node
I20260927 21:31:43.428567 18632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:43.430141 18632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:43.431401 18632 hybrid_clock.cc:648] HybridClock initialized: now 1790544703431354 us; error 28 us; skew 500 ppm
I20260927 21:31:43.431888 18632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:31:43 dist-test-slave-bxm0 krb5kdc[18619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:31:43.435282 18632 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:31:43.436889 18632 webserver.cc:460] Webserver started at http://127.0.97.62:34167/ using document root <none> and password file <none>
I20260927 21:31:43.437709 18632 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:43.437855 18632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:43.438179 18632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:31:43.441226 18632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "9cad8340f76240f2bd76b60cdd177f39"
format_stamp: "Formatted at 2026-09-27 21:31:43 on dist-test-slave-bxm0"
server_key: "9e11004fcb92e10d89ba95c324c4b502"
server_key_iv: "fbcd5bfb6b0c52c1f252514c80994235"
server_key_version: "encryptionkey@0"
I20260927 21:31:43.442127 18632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "9cad8340f76240f2bd76b60cdd177f39"
format_stamp: "Formatted at 2026-09-27 21:31:43 on dist-test-slave-bxm0"
server_key: "9e11004fcb92e10d89ba95c324c4b502"
server_key_iv: "fbcd5bfb6b0c52c1f252514c80994235"
server_key_version: "encryptionkey@0"
I20260927 21:31:43.447633 18632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 21:31:43.451226 18647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:43.452824 18632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:31:43.453037 18632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "9cad8340f76240f2bd76b60cdd177f39"
format_stamp: "Formatted at 2026-09-27 21:31:43 on dist-test-slave-bxm0"
server_key: "9e11004fcb92e10d89ba95c324c4b502"
server_key_iv: "fbcd5bfb6b0c52c1f252514c80994235"
server_key_version: "encryptionkey@0"
I20260927 21:31:43.453279 18632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:43.464820 18632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:43.469229 18632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:43.469588 18632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:43.488781 18632 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45027
I20260927 21:31:43.488799 18698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45027 every 8 connection(s)
I20260927 21:31:43.490288 18632 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:31:43.493269 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 18632
I20260927 21:31:43.493476 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:31:43.493947 388 external_mini_cluster.cc:1428] Setting key b43b2a65e1b8cb27a390bfe90eee9f28
I20260927 21:31:43.494412 18699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:31:43.509344 18699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cad8340f76240f2bd76b60cdd177f39: Bootstrap starting.
Sep 27 21:31:43 dist-test-slave-bxm0 krb5kdc[18619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:31:43.513891 18699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cad8340f76240f2bd76b60cdd177f39: Neither blocks nor log segments found. Creating new log.
I20260927 21:31:43.515331 18699 log.cc:824] T 00000000000000000000000000000000 P 9cad8340f76240f2bd76b60cdd177f39: Log is configured to *not* fsync() on all Append() calls
I20260927 21:31:43.519137 18699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cad8340f76240f2bd76b60cdd177f39: No bootstrap required, opened a new log
I20260927 21:31:43.520129 18702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:43.495934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33156 (local address 127.0.97.62:45027)
0927 21:31:43.496805 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:43.496823 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:43.496876 (+ 53us) server_negotiation.cc:407] Connection header received
0927 21:31:43.499378 (+ 2502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:43.499409 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:43.499984 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:43.500474 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:43.502043 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:43.504000 (+ 1957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:43.505429 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:43.506582 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:43.506786 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:43.511164 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:43.511178 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:43.511199 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:43.511255 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:43.515605 (+ 4350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:43.516503 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:43.516516 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:43.516526 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:43.516757 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:43.517256 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:43.517259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:43.517261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:43.517888 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:43.518102 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:43.518606 (+ 504us) server_negotiation.cc:299] Negotiation successful
0927 21:31:43.518944 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":361,"threads_started":1}
I20260927 21:31:43.527477 18699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cad8340f76240f2bd76b60cdd177f39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cad8340f76240f2bd76b60cdd177f39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45027 } }
I20260927 21:31:43.527899 18699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cad8340f76240f2bd76b60cdd177f39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:31:43.528007 18699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cad8340f76240f2bd76b60cdd177f39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cad8340f76240f2bd76b60cdd177f39, State: Initialized, Role: FOLLOWER
I20260927 21:31:43.528604 18699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cad8340f76240f2bd76b60cdd177f39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cad8340f76240f2bd76b60cdd177f39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45027 } }
I20260927 21:31:43.528851 18699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cad8340f76240f2bd76b60cdd177f39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:31:43.528996 18699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cad8340f76240f2bd76b60cdd177f39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:31:43.529176 18699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cad8340f76240f2bd76b60cdd177f39 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:31:43.531448 18699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cad8340f76240f2bd76b60cdd177f39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cad8340f76240f2bd76b60cdd177f39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45027 } }
I20260927 21:31:43.531949 18699 leader_election.cc:302] T 00000000000000000000000000000000 P 9cad8340f76240f2bd76b60cdd177f39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cad8340f76240f2bd76b60cdd177f39; no voters:
I20260927 21:31:43.532595 18699 leader_election.cc:288] T 00000000000000000000000000000000 P 9cad8340f76240f2bd76b60cdd177f39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:31:43.532732 18704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cad8340f76240f2bd76b60cdd177f39 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:31:43.533469 18704 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cad8340f76240f2bd76b60cdd177f39 [term 1 LEADER]: Becoming Leader. State: Replica: 9cad8340f76240f2bd76b60cdd177f39, State: Running, Role: LEADER
I20260927 21:31:43.533999 18704 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cad8340f76240f2bd76b60cdd177f39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cad8340f76240f2bd76b60cdd177f39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45027 } }
I20260927 21:31:43.534727 18699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cad8340f76240f2bd76b60cdd177f39 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:31:43.536803 18706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cad8340f76240f2bd76b60cdd177f39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cad8340f76240f2bd76b60cdd177f39. Latest consensus state: current_term: 1 leader_uuid: "9cad8340f76240f2bd76b60cdd177f39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cad8340f76240f2bd76b60cdd177f39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45027 } } }
I20260927 21:31:43.537175 18705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cad8340f76240f2bd76b60cdd177f39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cad8340f76240f2bd76b60cdd177f39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cad8340f76240f2bd76b60cdd177f39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45027 } } }
I20260927 21:31:43.537452 18706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cad8340f76240f2bd76b60cdd177f39 [sys.catalog]: This master's current role is: LEADER
I20260927 21:31:43.537462 18705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cad8340f76240f2bd76b60cdd177f39 [sys.catalog]: This master's current role is: LEADER
I20260927 21:31:43.543727 18710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:31:43.549193 18710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:31:43.558468 18710 catalog_manager.cc:1347] Generated new cluster ID: 99eaa64946244008ad23b84b759f8972
I20260927 21:31:43.558605 18710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:31:43.590260 18710 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:31:43.592176 18710 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:31:43.612576 18710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cad8340f76240f2bd76b60cdd177f39: Generated new TSK 0
I20260927 21:31:43.613530 18710 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 21:31:43.685029 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45027
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:31:44.045085 18725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:44.045459 18725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:44.045578 18725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:44.052881 18725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:44.052994 18725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:44.053107 18725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:31:44.063658 18725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:44.064913 18725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:44.066241 18725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:44.079628 18730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:44.079838 18731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:44.079859 18733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:44.080814 18725 server_base.cc:1034] running on GCE node
I20260927 21:31:44.081504 18725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:44.082543 18725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:44.083810 18725 hybrid_clock.cc:648] HybridClock initialized: now 1790544704083752 us; error 46 us; skew 500 ppm
I20260927 21:31:44.084170 18725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:31:44 dist-test-slave-bxm0 krb5kdc[18619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:31:44.087656 18725 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 21:31:44.089263 18725 webserver.cc:460] Webserver started at http://127.0.97.1:39515/ using document root <none> and password file <none>
I20260927 21:31:44.090070 18725 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:44.090221 18725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:44.090548 18725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:31:44.093708 18725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-0/data/instance:
uuid: "4e46f5e80e2e4cd08ca32896ac7b0316"
format_stamp: "Formatted at 2026-09-27 21:31:44 on dist-test-slave-bxm0"
server_key: "b112a23b12a046ed307546ac10707ced"
server_key_iv: "955c4facbdf08d36a79ae71a17de0eef"
server_key_version: "encryptionkey@0"
I20260927 21:31:44.094602 18725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4e46f5e80e2e4cd08ca32896ac7b0316"
format_stamp: "Formatted at 2026-09-27 21:31:44 on dist-test-slave-bxm0"
server_key: "b112a23b12a046ed307546ac10707ced"
server_key_iv: "955c4facbdf08d36a79ae71a17de0eef"
server_key_version: "encryptionkey@0"
I20260927 21:31:44.100004 18725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:31:44.103643 18740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:44.105391 18725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:31:44.105592 18725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "4e46f5e80e2e4cd08ca32896ac7b0316"
format_stamp: "Formatted at 2026-09-27 21:31:44 on dist-test-slave-bxm0"
server_key: "b112a23b12a046ed307546ac10707ced"
server_key_iv: "955c4facbdf08d36a79ae71a17de0eef"
server_key_version: "encryptionkey@0"
I20260927 21:31:44.105861 18725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:44.127353 18725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:44.131947 18725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:44.132254 18725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:44.133284 18725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:31:44.134874 18725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:31:44.134977 18725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:44.135128 18725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:31:44.135198 18725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:44.169016 18725 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34513
I20260927 21:31:44.169066 18852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34513 every 8 connection(s)
I20260927 21:31:44.170360 18725 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:31:44.175112 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 18725
I20260927 21:31:44.175308 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-0/wal/instance
I20260927 21:31:44.175854 388 external_mini_cluster.cc:1428] Setting key 9b388811388a6cc71a5f6c863a5a56c7
Sep 27 21:31:44 dist-test-slave-bxm0 krb5kdc[18619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:31:44.193110 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:44.172913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45145 (local address 127.0.97.62:45027)
0927 21:31:44.173373 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:44.173380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:44.174474 (+ 1094us) server_negotiation.cc:407] Connection header received
0927 21:31:44.177010 (+ 2536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:44.177016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:44.177092 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:44.177206 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:44.179220 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:44.179933 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:44.182478 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:44.182869 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:44.182967 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:44.187360 (+ 4393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:44.187366 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:44.187369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:44.187405 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:44.190074 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:44.190814 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:44.190824 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:44.190827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:44.190941 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:44.191378 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:44.191384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:44.191392 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:44.191642 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:44.191808 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:44.192508 (+ 700us) server_negotiation.cc:299] Negotiation successful
0927 21:31:44.192729 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":215,"threads_started":1}
I20260927 21:31:44.194180 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:44.173504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45027 (local address 127.0.97.1:45145)
0927 21:31:44.174284 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:44.174330 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:44.176696 (+ 2366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:44.177423 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:44.177432 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:44.178194 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:31:44.179026 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:44.179055 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:44.180089 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:44.180097 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:44.182298 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:44.182316 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:44.183103 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:44.183111 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:44.183461 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:44.184298 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:31:44.184337 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:31:44.187084 (+ 2747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:31:44.190217 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:31:44.190227 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:31:44.190245 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:31:44.190680 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:31:44.191052 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:31:44.191056 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:31:44.191060 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:31:44.191250 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:31:44.191770 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:31:44.191780 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:31:44.192132 (+ 352us) client_negotiation.cc:769] Sending connection context
0927 21:31:44.192401 (+ 269us) client_negotiation.cc:240] Negotiation successful
0927 21:31:44.192966 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":271,"threads_started":1}
I20260927 21:31:44.195923 18853 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45027
I20260927 21:31:44.196223 18853 heartbeater.cc:464] Registering TS with master...
I20260927 21:31:44.196949 18853 heartbeater.cc:511] Master 127.0.97.62:45027 requested a full tablet report, sending...
I20260927 21:31:44.199093 18664 ts_manager.cc:194] Registered new tserver with Master: 4e46f5e80e2e4cd08ca32896ac7b0316 (127.0.97.1:34513)
I20260927 21:31:44.200543 18664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45145
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 21:31:44.232219 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45027
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:31:44.562238 18859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:44.562538 18859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:44.562707 18859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:44.570215 18859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:44.570329 18859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:44.570505 18859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:31:44.580438 18859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:44.581805 18859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:44.583119 18859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:44.596237 18864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:44.596287 18867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:44.596325 18865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:44.597066 18859 server_base.cc:1034] running on GCE node
I20260927 21:31:44.597795 18859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:44.598938 18859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:44.600137 18859 hybrid_clock.cc:648] HybridClock initialized: now 1790544704600086 us; error 44 us; skew 500 ppm
I20260927 21:31:44.600487 18859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:31:44 dist-test-slave-bxm0 krb5kdc[18619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:31:44.603752 18859 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 21:31:44.605252 18859 webserver.cc:460] Webserver started at http://127.0.97.2:46145/ using document root <none> and password file <none>
I20260927 21:31:44.606029 18859 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:44.606132 18859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:44.606405 18859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:31:44.609200 18859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-1/data/instance:
uuid: "e7d46702d40441e58cec41f21af01da6"
format_stamp: "Formatted at 2026-09-27 21:31:44 on dist-test-slave-bxm0"
server_key: "281e6cc908499271ebb5429a844f2b7e"
server_key_iv: "c636daab9d94ba03a7ba9e46f10c8bf3"
server_key_version: "encryptionkey@0"
I20260927 21:31:44.610071 18859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e7d46702d40441e58cec41f21af01da6"
format_stamp: "Formatted at 2026-09-27 21:31:44 on dist-test-slave-bxm0"
server_key: "281e6cc908499271ebb5429a844f2b7e"
server_key_iv: "c636daab9d94ba03a7ba9e46f10c8bf3"
server_key_version: "encryptionkey@0"
I20260927 21:31:44.615427 18859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260927 21:31:44.619010 18874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:44.620388 18859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:31:44.620523 18859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "e7d46702d40441e58cec41f21af01da6"
format_stamp: "Formatted at 2026-09-27 21:31:44 on dist-test-slave-bxm0"
server_key: "281e6cc908499271ebb5429a844f2b7e"
server_key_iv: "c636daab9d94ba03a7ba9e46f10c8bf3"
server_key_version: "encryptionkey@0"
I20260927 21:31:44.620697 18859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:44.646462 18859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:44.650476 18859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:44.650722 18859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:44.651615 18859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:31:44.652868 18859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:31:44.652940 18859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:44.653021 18859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:31:44.653064 18859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:44.686303 18859 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33941
I20260927 21:31:44.686410 18986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33941 every 8 connection(s)
I20260927 21:31:44.687615 18859 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:31:44.697427 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 18859
I20260927 21:31:44.697641 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-1/wal/instance
I20260927 21:31:44.698237 388 external_mini_cluster.cc:1428] Setting key 023446e32263b85bc19f68b0ae650154
Sep 27 21:31:44 dist-test-slave-bxm0 krb5kdc[18619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:31:44.709174 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:44.690171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54653 (local address 127.0.97.62:45027)
0927 21:31:44.690364 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:44.690369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:44.691525 (+ 1156us) server_negotiation.cc:407] Connection header received
0927 21:31:44.693216 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:44.693220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:44.693293 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:44.693388 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:44.695104 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:44.695831 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:44.697366 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:44.698088 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:44.698225 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:44.702228 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:44.702237 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:44.702244 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:44.702291 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:44.705858 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:44.706625 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:44.706635 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:44.706642 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:44.706823 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:44.707220 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:44.707228 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:44.707234 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:44.707617 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:44.707782 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:44.708429 (+ 647us) server_negotiation.cc:299] Negotiation successful
0927 21:31:44.708694 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 21:31:44.709962 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:44.690644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45027 (local address 127.0.97.2:54653)
0927 21:31:44.691330 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:44.691366 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:44.692955 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:44.693588 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:44.693597 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:44.694210 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:31:44.694932 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:44.694954 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:44.695968 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:44.695972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:44.697199 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:44.697214 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:44.698310 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:44.698315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:44.698572 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:44.699292 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:31:44.699326 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:31:44.701998 (+ 2672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:31:44.706006 (+ 4008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:31:44.706014 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:31:44.706030 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:31:44.706492 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:31:44.706931 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:31:44.706935 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:31:44.706937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:31:44.707110 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:31:44.707733 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:31:44.707738 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:31:44.708070 (+ 332us) client_negotiation.cc:769] Sending connection context
0927 21:31:44.708307 (+ 237us) client_negotiation.cc:240] Negotiation successful
0927 21:31:44.708698 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":274,"threads_started":1}
I20260927 21:31:44.711598 18987 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45027
I20260927 21:31:44.711941 18987 heartbeater.cc:464] Registering TS with master...
I20260927 21:31:44.712805 18987 heartbeater.cc:511] Master 127.0.97.62:45027 requested a full tablet report, sending...
I20260927 21:31:44.714498 18664 ts_manager.cc:194] Registered new tserver with Master: e7d46702d40441e58cec41f21af01da6 (127.0.97.2:33941)
I20260927 21:31:44.715335 18664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54653
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 21:31:44.755298 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45027
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:31:45.085584 18992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:45.085953 18992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:45.086140 18992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:45.093472 18992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:45.093657 18992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:45.093854 18992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:31:45.103713 18992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:45.105072 18992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:45.106470 18992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:45.120558 18998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:45.120617 19000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:45.120649 18997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:45.121874 18992 server_base.cc:1034] running on GCE node
I20260927 21:31:45.122503 18992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:45.123430 18992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:45.124655 18992 hybrid_clock.cc:648] HybridClock initialized: now 1790544705124606 us; error 40 us; skew 500 ppm
I20260927 21:31:45.124989 18992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:31:45 dist-test-slave-bxm0 krb5kdc[18619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:31:45.128216 18992 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 21:31:45.129809 18992 webserver.cc:460] Webserver started at http://127.0.97.3:33279/ using document root <none> and password file <none>
I20260927 21:31:45.130497 18992 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:45.130625 18992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:45.130936 18992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:31:45.133865 18992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-2/data/instance:
uuid: "973e405bc56e4b2bae95a90bdffad4ab"
format_stamp: "Formatted at 2026-09-27 21:31:45 on dist-test-slave-bxm0"
server_key: "816f4823210ba9b26181bfc07b80f773"
server_key_iv: "6e2a2eb9e09dec021f19291dc03bdf89"
server_key_version: "encryptionkey@0"
I20260927 21:31:45.134672 18992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-2/wal/instance:
uuid: "973e405bc56e4b2bae95a90bdffad4ab"
format_stamp: "Formatted at 2026-09-27 21:31:45 on dist-test-slave-bxm0"
server_key: "816f4823210ba9b26181bfc07b80f773"
server_key_iv: "6e2a2eb9e09dec021f19291dc03bdf89"
server_key_version: "encryptionkey@0"
I20260927 21:31:45.140323 18992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:31:45.144654 19007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:45.146188 18992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 21:31:45.146322 18992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "973e405bc56e4b2bae95a90bdffad4ab"
format_stamp: "Formatted at 2026-09-27 21:31:45 on dist-test-slave-bxm0"
server_key: "816f4823210ba9b26181bfc07b80f773"
server_key_iv: "6e2a2eb9e09dec021f19291dc03bdf89"
server_key_version: "encryptionkey@0"
I20260927 21:31:45.146497 18992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:45.182741 18992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:45.187124 18992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:45.187367 18992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:45.188356 18992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:31:45.189810 18992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:31:45.189883 18992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:45.189976 18992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:31:45.190021 18992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:45.203847 18853 heartbeater.cc:503] Master 127.0.97.62:45027 was elected leader, sending a full tablet report...
I20260927 21:31:45.225558 18992 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34749
I20260927 21:31:45.225597 19119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34749 every 8 connection(s)
I20260927 21:31:45.226879 18992 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:31:45.230087 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 18992
I20260927 21:31:45.230270 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-2/wal/instance
I20260927 21:31:45.230911 388 external_mini_cluster.cc:1428] Setting key ab4562090b2183984bab95ea51aadd59
Sep 27 21:31:45 dist-test-slave-bxm0 krb5kdc[18619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:31:45.248735 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:45.229389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47803 (local address 127.0.97.62:45027)
0927 21:31:45.229993 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:45.230000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:45.231176 (+ 1176us) server_negotiation.cc:407] Connection header received
0927 21:31:45.233243 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:45.233249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:45.233329 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:45.233445 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:45.235293 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.236265 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.237809 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.238207 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.238299 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:45.241832 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:45.241837 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:45.241840 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:45.241874 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:45.245700 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:45.246687 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:45.246692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:45.246694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:45.246836 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:45.247234 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:45.247237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:45.247239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:45.247503 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:45.247612 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:45.248279 (+ 667us) server_negotiation.cc:299] Negotiation successful
0927 21:31:45.248468 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":340,"threads_started":1}
I20260927 21:31:45.249800 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:45.230077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45027 (local address 127.0.97.3:47803)
0927 21:31:45.230970 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:45.231035 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:45.232935 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:45.233718 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:45.233729 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:45.234413 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:31:45.235112 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:45.235130 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.236399 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.236402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:45.237669 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:45.237687 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.238413 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.238418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:45.238622 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:45.239328 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:31:45.239354 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:31:45.241539 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:31:45.245863 (+ 4324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:31:45.245874 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:31:45.245894 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:31:45.246552 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:31:45.246937 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:31:45.246940 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:31:45.246943 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:31:45.247135 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:31:45.247607 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:31:45.247614 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:31:45.247957 (+ 343us) client_negotiation.cc:769] Sending connection context
0927 21:31:45.248186 (+ 229us) client_negotiation.cc:240] Negotiation successful
0927 21:31:45.248608 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":350,"threads_started":1}
I20260927 21:31:45.251183 19120 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45027
I20260927 21:31:45.251508 19120 heartbeater.cc:464] Registering TS with master...
I20260927 21:31:45.252207 19120 heartbeater.cc:511] Master 127.0.97.62:45027 requested a full tablet report, sending...
I20260927 21:31:45.253546 18664 ts_manager.cc:194] Registered new tserver with Master: 973e405bc56e4b2bae95a90bdffad4ab (127.0.97.3:34749)
I20260927 21:31:45.254367 18664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47803
I20260927 21:31:45.258622 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:31:45.275125 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:45.262603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33164 (local address 127.0.97.62:45027)
0927 21:31:45.262803 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:45.262826 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:45.263307 (+ 481us) server_negotiation.cc:407] Connection header received
0927 21:31:45.263449 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:45.263452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:45.263526 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:45.263611 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:45.264929 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.265704 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.267021 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.267387 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.267474 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:45.269941 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:45.269945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:45.269949 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:45.269981 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:45.272631 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:45.273414 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:45.273417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:45.273419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:45.273528 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:45.273955 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:45.273958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:45.273960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:45.274256 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:45.274444 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:45.274768 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 21:31:45.274900 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:31:45.278995 18664 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:31:45.285837 18664 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:31:45.320690 19136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:45.298626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52892 (local address 127.0.97.3:34749)
0927 21:31:45.310382 (+ 11756us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:45.310391 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:45.311332 (+ 941us) server_negotiation.cc:407] Connection header received
0927 21:31:45.311463 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:45.311475 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:45.311762 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:45.311970 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:45.313942 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.315639 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.318295 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.319476 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.319606 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:45.319876 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:45.320160 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 21:31:45.320342 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11572,"thread_start_us":11405,"threads_started":1}
I20260927 21:31:45.320780 19133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:45.298206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34749 (local address 127.0.0.1:52892)
0927 21:31:45.310814 (+ 12608us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:45.310856 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:45.311101 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:45.312350 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:45.312361 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:45.312404 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:45.313033 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:45.313053 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.315776 (+ 2723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.315781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:45.318100 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:45.318115 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.319661 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.319667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:45.319886 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:45.319947 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 21:31:45.320077 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 21:31:45.320296 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12420,"mutex_wait_us":6,"spinlock_wait_cycles":6656,"thread_start_us":11334,"threads_started":1}
I20260927 21:31:45.322031 19135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:45.297872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34513 (local address 127.0.0.1:37292)
0927 21:31:45.310756 (+ 12884us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:45.310819 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:45.311103 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:45.312603 (+ 1500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:45.312611 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:45.312646 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:45.313591 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:45.313630 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.316394 (+ 2764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.316399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:45.318580 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:45.318592 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.320629 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.320635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:45.321260 (+ 625us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:45.321370 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 21:31:45.321443 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 21:31:45.321573 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11841,"mutex_wait_us":64,"spinlock_wait_cycles":11648,"thread_start_us":12575,"threads_started":1}
I20260927 21:31:45.322615 19134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:45.298115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37292 (local address 127.0.97.1:34513)
0927 21:31:45.311887 (+ 13772us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:45.311900 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:45.311936 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:31:45.312050 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:45.312060 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:45.312345 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:45.313004 (+ 659us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:45.313949 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.316246 (+ 2297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.318747 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.320143 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.320268 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:45.320486 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:45.322257 (+ 1771us) server_negotiation.cc:299] Negotiation successful
0927 21:31:45.322396 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13550,"thread_start_us":13351,"threads_started":1}
I20260927 21:31:45.322813 19137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:45.310955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49540 (local address 127.0.97.2:33941)
0927 21:31:45.313593 (+ 2638us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:45.313602 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:45.314205 (+ 603us) server_negotiation.cc:407] Connection header received
0927 21:31:45.314308 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:45.314316 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:45.314531 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:45.314706 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:45.315502 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.316808 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.319363 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.320766 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.320934 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:45.321100 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:45.322341 (+ 1241us) server_negotiation.cc:299] Negotiation successful
0927 21:31:45.322513 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2443,"thread_start_us":2274,"threads_started":1}
I20260927 21:31:45.323040 19138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:45.311839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33941 (local address 127.0.0.1:49540)
0927 21:31:45.313955 (+ 2116us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:45.313977 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:45.314091 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:45.314959 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:45.314964 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:45.314986 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:45.315319 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:45.315330 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.317014 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.317019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:45.319192 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:45.319206 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.321271 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.321278 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:45.321487 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:45.321521 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 21:31:45.321627 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 21:31:45.321751 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1977,"mutex_wait_us":34,"thread_start_us":1760,"threads_started":1}
I20260927 21:31:45.325227 19055 tablet_service.cc:1467] Processing CreateTablet for tablet 91b4208979394b74855ae4b9d394da39 (DEFAULT_TABLE table=test-table [id=4e0e141fde0044be9b3868bd7fcaa50c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:31:45.326340 18922 tablet_service.cc:1467] Processing CreateTablet for tablet 91b4208979394b74855ae4b9d394da39 (DEFAULT_TABLE table=test-table [id=4e0e141fde0044be9b3868bd7fcaa50c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:31:45.326390 18788 tablet_service.cc:1467] Processing CreateTablet for tablet 91b4208979394b74855ae4b9d394da39 (DEFAULT_TABLE table=test-table [id=4e0e141fde0044be9b3868bd7fcaa50c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:31:45.326903 19055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91b4208979394b74855ae4b9d394da39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:31:45.327431 18922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91b4208979394b74855ae4b9d394da39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:31:45.327682 18788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91b4208979394b74855ae4b9d394da39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:31:45.338771 19139 tablet_bootstrap.cc:492] T 91b4208979394b74855ae4b9d394da39 P e7d46702d40441e58cec41f21af01da6: Bootstrap starting.
I20260927 21:31:45.342723 19139 tablet_bootstrap.cc:654] T 91b4208979394b74855ae4b9d394da39 P e7d46702d40441e58cec41f21af01da6: Neither blocks nor log segments found. Creating new log.
I20260927 21:31:45.343845 19139 log.cc:824] T 91b4208979394b74855ae4b9d394da39 P e7d46702d40441e58cec41f21af01da6: Log is configured to *not* fsync() on all Append() calls
I20260927 21:31:45.346272 19141 tablet_bootstrap.cc:492] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab: Bootstrap starting.
I20260927 21:31:45.346894 19140 tablet_bootstrap.cc:492] T 91b4208979394b74855ae4b9d394da39 P 4e46f5e80e2e4cd08ca32896ac7b0316: Bootstrap starting.
I20260927 21:31:45.347189 19139 tablet_bootstrap.cc:492] T 91b4208979394b74855ae4b9d394da39 P e7d46702d40441e58cec41f21af01da6: No bootstrap required, opened a new log
I20260927 21:31:45.347476 19139 ts_tablet_manager.cc:1397] T 91b4208979394b74855ae4b9d394da39 P e7d46702d40441e58cec41f21af01da6: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 21:31:45.350108 19141 tablet_bootstrap.cc:654] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab: Neither blocks nor log segments found. Creating new log.
I20260927 21:31:45.350319 19140 tablet_bootstrap.cc:654] T 91b4208979394b74855ae4b9d394da39 P 4e46f5e80e2e4cd08ca32896ac7b0316: Neither blocks nor log segments found. Creating new log.
I20260927 21:31:45.351401 19141 log.cc:824] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab: Log is configured to *not* fsync() on all Append() calls
I20260927 21:31:45.351434 19140 log.cc:824] T 91b4208979394b74855ae4b9d394da39 P 4e46f5e80e2e4cd08ca32896ac7b0316: Log is configured to *not* fsync() on all Append() calls
I20260927 21:31:45.354303 19141 tablet_bootstrap.cc:492] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab: No bootstrap required, opened a new log
I20260927 21:31:45.354511 19140 tablet_bootstrap.cc:492] T 91b4208979394b74855ae4b9d394da39 P 4e46f5e80e2e4cd08ca32896ac7b0316: No bootstrap required, opened a new log
I20260927 21:31:45.354593 19141 ts_tablet_manager.cc:1397] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 21:31:45.354765 19140 ts_tablet_manager.cc:1397] T 91b4208979394b74855ae4b9d394da39 P 4e46f5e80e2e4cd08ca32896ac7b0316: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
I20260927 21:31:45.358038 19139 raft_consensus.cc:348] T 91b4208979394b74855ae4b9d394da39 P e7d46702d40441e58cec41f21af01da6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e46f5e80e2e4cd08ca32896ac7b0316" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34513 } } peers { permanent_uuid: "973e405bc56e4b2bae95a90bdffad4ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34749 } } peers { permanent_uuid: "e7d46702d40441e58cec41f21af01da6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33941 } }
I20260927 21:31:45.358520 19139 raft_consensus.cc:374] T 91b4208979394b74855ae4b9d394da39 P e7d46702d40441e58cec41f21af01da6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:31:45.358755 19139 raft_consensus.cc:729] T 91b4208979394b74855ae4b9d394da39 P e7d46702d40441e58cec41f21af01da6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7d46702d40441e58cec41f21af01da6, State: Initialized, Role: FOLLOWER
I20260927 21:31:45.359413 19139 consensus_queue.cc:260] T 91b4208979394b74855ae4b9d394da39 P e7d46702d40441e58cec41f21af01da6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e46f5e80e2e4cd08ca32896ac7b0316" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34513 } } peers { permanent_uuid: "973e405bc56e4b2bae95a90bdffad4ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34749 } } peers { permanent_uuid: "e7d46702d40441e58cec41f21af01da6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33941 } }
I20260927 21:31:45.360628 18987 heartbeater.cc:503] Master 127.0.97.62:45027 was elected leader, sending a full tablet report...
I20260927 21:31:45.361227 19139 ts_tablet_manager.cc:1428] T 91b4208979394b74855ae4b9d394da39 P e7d46702d40441e58cec41f21af01da6: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260927 21:31:45.365257 19140 raft_consensus.cc:348] T 91b4208979394b74855ae4b9d394da39 P 4e46f5e80e2e4cd08ca32896ac7b0316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e46f5e80e2e4cd08ca32896ac7b0316" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34513 } } peers { permanent_uuid: "973e405bc56e4b2bae95a90bdffad4ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34749 } } peers { permanent_uuid: "e7d46702d40441e58cec41f21af01da6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33941 } }
I20260927 21:31:45.365800 19140 raft_consensus.cc:374] T 91b4208979394b74855ae4b9d394da39 P 4e46f5e80e2e4cd08ca32896ac7b0316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:31:45.365931 19140 raft_consensus.cc:729] T 91b4208979394b74855ae4b9d394da39 P 4e46f5e80e2e4cd08ca32896ac7b0316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e46f5e80e2e4cd08ca32896ac7b0316, State: Initialized, Role: FOLLOWER
I20260927 21:31:45.366061 19141 raft_consensus.cc:348] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e46f5e80e2e4cd08ca32896ac7b0316" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34513 } } peers { permanent_uuid: "973e405bc56e4b2bae95a90bdffad4ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34749 } } peers { permanent_uuid: "e7d46702d40441e58cec41f21af01da6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33941 } }
I20260927 21:31:45.366478 19141 raft_consensus.cc:374] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:31:45.366571 19141 raft_consensus.cc:729] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 973e405bc56e4b2bae95a90bdffad4ab, State: Initialized, Role: FOLLOWER
I20260927 21:31:45.366598 19140 consensus_queue.cc:260] T 91b4208979394b74855ae4b9d394da39 P 4e46f5e80e2e4cd08ca32896ac7b0316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e46f5e80e2e4cd08ca32896ac7b0316" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34513 } } peers { permanent_uuid: "973e405bc56e4b2bae95a90bdffad4ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34749 } } peers { permanent_uuid: "e7d46702d40441e58cec41f21af01da6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33941 } }
I20260927 21:31:45.367172 19141 consensus_queue.cc:260] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e46f5e80e2e4cd08ca32896ac7b0316" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34513 } } peers { permanent_uuid: "973e405bc56e4b2bae95a90bdffad4ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34749 } } peers { permanent_uuid: "e7d46702d40441e58cec41f21af01da6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33941 } }
I20260927 21:31:45.368645 19140 ts_tablet_manager.cc:1428] T 91b4208979394b74855ae4b9d394da39 P 4e46f5e80e2e4cd08ca32896ac7b0316: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260927 21:31:45.374202 19120 heartbeater.cc:503] Master 127.0.97.62:45027 was elected leader, sending a full tablet report...
I20260927 21:31:45.374923 19141 ts_tablet_manager.cc:1428] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab: Time spent starting tablet: real 0.020s user 0.013s sys 0.005s
W20260927 21:31:45.422672 18854 tablet.cc:2369] T 91b4208979394b74855ae4b9d394da39 P 4e46f5e80e2e4cd08ca32896ac7b0316: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:31:45.435565 19147 raft_consensus.cc:482] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:31:45.435987 19147 raft_consensus.cc:504] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e46f5e80e2e4cd08ca32896ac7b0316" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34513 } } peers { permanent_uuid: "973e405bc56e4b2bae95a90bdffad4ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34749 } } peers { permanent_uuid: "e7d46702d40441e58cec41f21af01da6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33941 } }
I20260927 21:31:45.437795 19147 leader_election.cc:288] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e46f5e80e2e4cd08ca32896ac7b0316 (127.0.97.1:34513), e7d46702d40441e58cec41f21af01da6 (127.0.97.2:33941)
W20260927 21:31:45.439756 18988 tablet.cc:2369] T 91b4208979394b74855ae4b9d394da39 P e7d46702d40441e58cec41f21af01da6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:31:45.445255 19137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:45.438239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40367 (local address 127.0.97.2:33941)
0927 21:31:45.438485 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:45.438492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:45.438722 (+ 230us) server_negotiation.cc:407] Connection header received
0927 21:31:45.438942 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:45.438948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:45.439039 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:45.439140 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:45.439623 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.440733 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.443155 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.444351 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.444491 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:45.444597 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:45.444870 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 21:31:45.445026 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260927 21:31:45.445339 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:45.438170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33941 (local address 127.0.97.3:40367)
0927 21:31:45.438585 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:45.438614 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:45.438781 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:45.439178 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:45.439182 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:45.439224 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:45.439451 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:45.439463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.440884 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.440888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:45.442996 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:45.443010 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.444508 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.444519 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:45.444658 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:45.444684 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 21:31:45.444816 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 21:31:45.445022 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":1}
I20260927 21:31:45.445654 19134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:45.438276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38727 (local address 127.0.97.1:34513)
0927 21:31:45.438462 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:45.438478 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:45.439130 (+ 652us) server_negotiation.cc:407] Connection header received
0927 21:31:45.439346 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:45.439352 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:45.439432 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:45.439536 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:45.440145 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.441068 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.443385 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.444620 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.444744 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:45.444863 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:45.445245 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 21:31:45.445380 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 21:31:45.445802 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:45.438204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34513 (local address 127.0.97.3:38727)
0927 21:31:45.438982 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:45.438999 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:45.439153 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:45.439570 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:45.439574 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:45.439600 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:45.439976 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:45.439988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.441212 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.441218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:45.443242 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:45.443251 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.444846 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.444851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:45.445068 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:45.445099 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 21:31:45.445185 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 21:31:45.445297 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":30,"thread_start_us":416,"threads_started":1}
I20260927 21:31:45.446446 18942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b4208979394b74855ae4b9d394da39" candidate_uuid: "973e405bc56e4b2bae95a90bdffad4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7d46702d40441e58cec41f21af01da6" is_pre_election: true
I20260927 21:31:45.446707 18808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b4208979394b74855ae4b9d394da39" candidate_uuid: "973e405bc56e4b2bae95a90bdffad4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e46f5e80e2e4cd08ca32896ac7b0316" is_pre_election: true
I20260927 21:31:45.446940 18942 raft_consensus.cc:2454] T 91b4208979394b74855ae4b9d394da39 P e7d46702d40441e58cec41f21af01da6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 973e405bc56e4b2bae95a90bdffad4ab in term 0.
I20260927 21:31:45.447255 18808 raft_consensus.cc:2454] T 91b4208979394b74855ae4b9d394da39 P 4e46f5e80e2e4cd08ca32896ac7b0316 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 973e405bc56e4b2bae95a90bdffad4ab in term 0.
I20260927 21:31:45.447659 19008 leader_election.cc:302] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 973e405bc56e4b2bae95a90bdffad4ab, e7d46702d40441e58cec41f21af01da6; no voters:
I20260927 21:31:45.448071 19147 raft_consensus.cc:2781] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:31:45.448228 19147 raft_consensus.cc:482] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:31:45.448339 19147 raft_consensus.cc:3037] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:31:45.450676 19147 raft_consensus.cc:504] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e46f5e80e2e4cd08ca32896ac7b0316" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34513 } } peers { permanent_uuid: "973e405bc56e4b2bae95a90bdffad4ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34749 } } peers { permanent_uuid: "e7d46702d40441e58cec41f21af01da6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33941 } }
I20260927 21:31:45.451287 19147 leader_election.cc:288] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab [CANDIDATE]: Term 1 election: Requested vote from peers 4e46f5e80e2e4cd08ca32896ac7b0316 (127.0.97.1:34513), e7d46702d40441e58cec41f21af01da6 (127.0.97.2:33941)
I20260927 21:31:45.451764 18808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b4208979394b74855ae4b9d394da39" candidate_uuid: "973e405bc56e4b2bae95a90bdffad4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e46f5e80e2e4cd08ca32896ac7b0316"
I20260927 21:31:45.451794 18942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b4208979394b74855ae4b9d394da39" candidate_uuid: "973e405bc56e4b2bae95a90bdffad4ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7d46702d40441e58cec41f21af01da6"
I20260927 21:31:45.451977 18808 raft_consensus.cc:3037] T 91b4208979394b74855ae4b9d394da39 P 4e46f5e80e2e4cd08ca32896ac7b0316 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:31:45.452009 18942 raft_consensus.cc:3037] T 91b4208979394b74855ae4b9d394da39 P e7d46702d40441e58cec41f21af01da6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:31:45.454277 18808 raft_consensus.cc:2454] T 91b4208979394b74855ae4b9d394da39 P 4e46f5e80e2e4cd08ca32896ac7b0316 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 973e405bc56e4b2bae95a90bdffad4ab in term 1.
I20260927 21:31:45.454303 18942 raft_consensus.cc:2454] T 91b4208979394b74855ae4b9d394da39 P e7d46702d40441e58cec41f21af01da6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 973e405bc56e4b2bae95a90bdffad4ab in term 1.
I20260927 21:31:45.454751 19008 leader_election.cc:302] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4e46f5e80e2e4cd08ca32896ac7b0316, 973e405bc56e4b2bae95a90bdffad4ab, e7d46702d40441e58cec41f21af01da6; no voters:
I20260927 21:31:45.455102 19147 raft_consensus.cc:2781] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:31:45.455579 19147 raft_consensus.cc:686] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab [term 1 LEADER]: Becoming Leader. State: Replica: 973e405bc56e4b2bae95a90bdffad4ab, State: Running, Role: LEADER
I20260927 21:31:45.456079 19147 consensus_queue.cc:237] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e46f5e80e2e4cd08ca32896ac7b0316" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34513 } } peers { permanent_uuid: "973e405bc56e4b2bae95a90bdffad4ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34749 } } peers { permanent_uuid: "e7d46702d40441e58cec41f21af01da6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33941 } }
I20260927 21:31:45.461087 18664 catalog_manager.cc:5495] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 973e405bc56e4b2bae95a90bdffad4ab (127.0.97.3). New cstate: current_term: 1 leader_uuid: "973e405bc56e4b2bae95a90bdffad4ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e46f5e80e2e4cd08ca32896ac7b0316" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "973e405bc56e4b2bae95a90bdffad4ab" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7d46702d40441e58cec41f21af01da6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33941 } health_report { overall_health: UNKNOWN } } }
W20260927 21:31:45.478549 19121 tablet.cc:2369] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:31:45.484925 19136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:45.479818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52908 (local address 127.0.97.3:34749)
0927 21:31:45.479999 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:45.480004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:45.480127 (+ 123us) server_negotiation.cc:407] Connection header received
0927 21:31:45.480306 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:45.480309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:45.480395 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:45.480522 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:31:45.480856 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.481561 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.482947 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.483349 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.483440 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:45.483929 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:31:45.484419 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:31:45.484548 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:45.484684 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 21:31:45.484725 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:31:45.493227 18942 raft_consensus.cc:1261] T 91b4208979394b74855ae4b9d394da39 P e7d46702d40441e58cec41f21af01da6 [term 1 FOLLOWER]: Refusing update from remote peer 973e405bc56e4b2bae95a90bdffad4ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:31:45.493412 18808 raft_consensus.cc:1261] T 91b4208979394b74855ae4b9d394da39 P 4e46f5e80e2e4cd08ca32896ac7b0316 [term 1 FOLLOWER]: Refusing update from remote peer 973e405bc56e4b2bae95a90bdffad4ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:31:45.494360 19149 consensus_queue.cc:1035] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e46f5e80e2e4cd08ca32896ac7b0316" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:31:45.494782 19147 consensus_queue.cc:1035] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7d46702d40441e58cec41f21af01da6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:31:45.508001 19152 mvcc.cc:205] Tried to move back new op lower bound from 7334071113689939968 to 7334071113552396288. Current Snapshot: MvccSnapshot[applied={T|T < 7334071113689939968}]
I20260927 21:31:45.513103 19154 mvcc.cc:205] Tried to move back new op lower bound from 7334071113689939968 to 7334071113552396288. Current Snapshot: MvccSnapshot[applied={T|T < 7334071113689939968}]
I20260927 21:31:45.522017 18663 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:31:45.522337 18663 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:31:45.526290 18663 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9cad8340f76240f2bd76b60cdd177f39: Sending DeleteTablet for 3 replicas of tablet 91b4208979394b74855ae4b9d394da39
I20260927 21:31:45.527410 18788 tablet_service.cc:1514] Processing DeleteTablet for tablet 91b4208979394b74855ae4b9d394da39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:31:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37292
I20260927 21:31:45.527567 18922 tablet_service.cc:1514] Processing DeleteTablet for tablet 91b4208979394b74855ae4b9d394da39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:31:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49540
I20260927 21:31:45.527590 19055 tablet_service.cc:1514] Processing DeleteTablet for tablet 91b4208979394b74855ae4b9d394da39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:31:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52892
I20260927 21:31:45.532269 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 18725
I20260927 21:31:45.543929 19162 tablet_replica.cc:330] T 91b4208979394b74855ae4b9d394da39 P e7d46702d40441e58cec41f21af01da6: stopping tablet replica
I20260927 21:31:45.544137 19163 tablet_replica.cc:330] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab: stopping tablet replica
I20260927 21:31:45.544603 19163 raft_consensus.cc:2229] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:31:45.544637 19162 raft_consensus.cc:2229] T 91b4208979394b74855ae4b9d394da39 P e7d46702d40441e58cec41f21af01da6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:31:45.545305 19162 raft_consensus.cc:2258] T 91b4208979394b74855ae4b9d394da39 P e7d46702d40441e58cec41f21af01da6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:31:45.545346 19163 raft_consensus.cc:2258] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:31:45.546934 19161 tablet_replica.cc:330] T 91b4208979394b74855ae4b9d394da39 P 4e46f5e80e2e4cd08ca32896ac7b0316: stopping tablet replica
I20260927 21:31:45.547292 19161 raft_consensus.cc:2229] T 91b4208979394b74855ae4b9d394da39 P 4e46f5e80e2e4cd08ca32896ac7b0316 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:31:45.547721 19161 raft_consensus.cc:2258] T 91b4208979394b74855ae4b9d394da39 P 4e46f5e80e2e4cd08ca32896ac7b0316 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:31:45.547889 19163 ts_tablet_manager.cc:1905] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:31:45.551539 19161 ts_tablet_manager.cc:1905] T 91b4208979394b74855ae4b9d394da39 P 4e46f5e80e2e4cd08ca32896ac7b0316: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:31:45.554126 19162 ts_tablet_manager.cc:1905] T 91b4208979394b74855ae4b9d394da39 P e7d46702d40441e58cec41f21af01da6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:31:45.555544 19163 ts_tablet_manager.cc:1918] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:31:45.555713 19163 log.cc:1196] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-2/wal/wals/91b4208979394b74855ae4b9d394da39
I20260927 21:31:45.556257 19163 ts_tablet_manager.cc:1939] T 91b4208979394b74855ae4b9d394da39 P 973e405bc56e4b2bae95a90bdffad4ab: Deleting consensus metadata
I20260927 21:31:45.558344 18648 catalog_manager.cc:4841] TS 973e405bc56e4b2bae95a90bdffad4ab (127.0.97.3:34749): tablet 91b4208979394b74855ae4b9d394da39 (table test-table [id=4e0e141fde0044be9b3868bd7fcaa50c]) successfully deleted
Sep 27 21:31:45 dist-test-slave-bxm0 krb5kdc[18619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 21:31:45.562605 19162 ts_tablet_manager.cc:1918] T 91b4208979394b74855ae4b9d394da39 P e7d46702d40441e58cec41f21af01da6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:31:45.562783 19162 log.cc:1196] T 91b4208979394b74855ae4b9d394da39 P e7d46702d40441e58cec41f21af01da6: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-1/wal/wals/91b4208979394b74855ae4b9d394da39
I20260927 21:31:45.563365 19162 ts_tablet_manager.cc:1939] T 91b4208979394b74855ae4b9d394da39 P e7d46702d40441e58cec41f21af01da6: Deleting consensus metadata
I20260927 21:31:45.563855 19161 ts_tablet_manager.cc:1918] T 91b4208979394b74855ae4b9d394da39 P 4e46f5e80e2e4cd08ca32896ac7b0316: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:31:45.564016 19161 log.cc:1196] T 91b4208979394b74855ae4b9d394da39 P 4e46f5e80e2e4cd08ca32896ac7b0316: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790544390748963-388-0/minicluster-data/ts-0/wal/wals/91b4208979394b74855ae4b9d394da39
I20260927 21:31:45.564561 19161 ts_tablet_manager.cc:1939] T 91b4208979394b74855ae4b9d394da39 P 4e46f5e80e2e4cd08ca32896ac7b0316: Deleting consensus metadata
I20260927 21:31:45.565331 18648 catalog_manager.cc:4841] TS e7d46702d40441e58cec41f21af01da6 (127.0.97.2:33941): tablet 91b4208979394b74855ae4b9d394da39 (table test-table [id=4e0e141fde0044be9b3868bd7fcaa50c]) successfully deleted
I20260927 21:31:45.566275 18650 catalog_manager.cc:4841] TS 4e46f5e80e2e4cd08ca32896ac7b0316 (127.0.97.1:34513): tablet 91b4208979394b74855ae4b9d394da39 (table test-table [id=4e0e141fde0044be9b3868bd7fcaa50c]) successfully deleted
I20260927 21:31:45.568318 19134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:45.533531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37298 (local address 127.0.97.1:34513)
0927 21:31:45.533754 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:45.533761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:45.548417 (+ 14656us) server_negotiation.cc:407] Connection header received
0927 21:31:45.548625 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:45.548631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:45.548725 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:45.548824 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:45.550749 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.551863 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.553836 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.554423 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.554524 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:45.560606 (+ 6082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:45.560626 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:45.560647 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:45.560692 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:45.564976 (+ 4284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:45.566047 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:45.566058 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:45.566061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:45.566261 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:45.566729 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:45.566734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:45.566738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:45.567100 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:45.567406 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:45.567733 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 21:31:45.567961 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 21:31:45.569064 18848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:45.662459 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 18725
I20260927 21:31:45.682790 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 18859
Sep 27 21:31:45 dist-test-slave-bxm0 krb5kdc[18619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 21:31:45.703012 19137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:45.683630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49554 (local address 127.0.97.2:33941)
0927 21:31:45.683829 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:45.683835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:45.683854 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:31:45.684016 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:45.684022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:45.684110 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:45.684224 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:45.685504 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.686717 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.688589 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.689245 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.689380 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:45.694965 (+ 5585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:45.694981 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:45.695001 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:45.695049 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:45.699148 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:45.700075 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:45.700086 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:45.700091 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:45.700254 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:45.700662 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:45.700667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:45.700670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:45.701186 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:45.701479 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:45.702431 (+ 952us) server_negotiation.cc:299] Negotiation successful
0927 21:31:45.702649 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 21:31:45.703723 18982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:45.793145 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 18859
I20260927 21:31:45.810684 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 18992
Sep 27 21:31:45 dist-test-slave-bxm0 krb5kdc[18619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 21:31:45.826853 19136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:45.811479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52918 (local address 127.0.97.3:34749)
0927 21:31:45.811645 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:45.811653 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:45.811713 (+ 60us) server_negotiation.cc:407] Connection header received
0927 21:31:45.811944 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:45.811949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:45.812033 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:45.812151 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:45.813462 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.814351 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.815618 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:45.816081 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:45.816226 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:45.820636 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:45.820651 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:45.820670 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:45.820705 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:45.823855 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:45.824568 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:45.824576 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:45.824578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:45.824740 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:45.825131 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:45.825136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:45.825141 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:45.825499 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:45.826105 (+ 606us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:45.826271 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 21:31:45.826514 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 21:31:45.827551 19115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:45.950626 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 18992
I20260927 21:31:45.970531 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 18632
I20260927 21:31:45.971585 18694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:46.083326 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 18632
2026-09-27T21:31:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4400 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T21:31:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:31:46Z Disabled control of system clock
I20260927 21:31:46.160676 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35519
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:31:46.492832 19172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:46.493136 19172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:46.493218 19172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:46.500540 19172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:31:46.500669 19172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:46.500707 19172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:46.500758 19172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:31:46.500815 19172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:31:46.510916 19172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35519
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:46.512166 19172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:46.514019 19172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:46.525959 19178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:46.525959 19177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:46.526314 19180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:46.526970 19172 server_base.cc:1034] running on GCE node
I20260927 21:31:46.527819 19172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:46.529289 19172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:46.530491 19172 hybrid_clock.cc:648] HybridClock initialized: now 1790544706530448 us; error 33 us; skew 500 ppm
I20260927 21:31:46.530864 19172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:31:46.539148 19172 webserver.cc:460] Webserver started at http://127.0.97.62:34987/ using document root <none> and password file <none>
I20260927 21:31:46.539913 19172 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:46.540036 19172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:46.540385 19172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:31:46.543462 19172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "9ebb7028eec54c56a8659f91e4a8f644"
format_stamp: "Formatted at 2026-09-27 21:31:46 on dist-test-slave-bxm0"
server_key: "5bdc7189fc7ed9bb82b4b00fdcf2ba0d"
server_key_iv: "1552bdd5a5c2b64d8e8ccda9df850f91"
server_key_version: "encryptionkey@0"
I20260927 21:31:46.544281 19172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "9ebb7028eec54c56a8659f91e4a8f644"
format_stamp: "Formatted at 2026-09-27 21:31:46 on dist-test-slave-bxm0"
server_key: "5bdc7189fc7ed9bb82b4b00fdcf2ba0d"
server_key_iv: "1552bdd5a5c2b64d8e8ccda9df850f91"
server_key_version: "encryptionkey@0"
I20260927 21:31:46.549454 19172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 21:31:46.553005 19186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:46.554455 19172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:31:46.554643 19172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "9ebb7028eec54c56a8659f91e4a8f644"
format_stamp: "Formatted at 2026-09-27 21:31:46 on dist-test-slave-bxm0"
server_key: "5bdc7189fc7ed9bb82b4b00fdcf2ba0d"
server_key_iv: "1552bdd5a5c2b64d8e8ccda9df850f91"
server_key_version: "encryptionkey@0"
I20260927 21:31:46.554888 19172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:46.585453 19172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:46.586391 19172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:46.586697 19172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:46.605161 19172 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35519
I20260927 21:31:46.605260 19237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35519 every 8 connection(s)
I20260927 21:31:46.606593 19172 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:31:46.610384 19238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:31:46.615725 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 19172
I20260927 21:31:46.615924 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:31:46.616348 388 external_mini_cluster.cc:1428] Setting key 71f65ba3d654f391a89e9a25f6d89027
I20260927 21:31:46.621776 19238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ebb7028eec54c56a8659f91e4a8f644: Bootstrap starting.
I20260927 21:31:46.624928 19238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ebb7028eec54c56a8659f91e4a8f644: Neither blocks nor log segments found. Creating new log.
I20260927 21:31:46.626040 19238 log.cc:824] T 00000000000000000000000000000000 P 9ebb7028eec54c56a8659f91e4a8f644: Log is configured to *not* fsync() on all Append() calls
I20260927 21:31:46.629338 19238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ebb7028eec54c56a8659f91e4a8f644: No bootstrap required, opened a new log
I20260927 21:31:46.630345 19241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:46.618369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40100 (local address 127.0.97.62:35519)
0927 21:31:46.619419 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:46.619430 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:46.619474 (+ 44us) server_negotiation.cc:407] Connection header received
0927 21:31:46.621328 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:46.621356 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:46.621944 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:46.622358 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:46.622943 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:46.624264 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:46.625777 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:46.626602 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:46.626822 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:46.627972 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:46.627980 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:46.627995 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:31:46.628518 (+ 523us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:46.628599 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:31:46.628620 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:46.628711 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:46.628947 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 21:31:46.629289 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":367,"threads_started":1}
I20260927 21:31:46.636806 19238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ebb7028eec54c56a8659f91e4a8f644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ebb7028eec54c56a8659f91e4a8f644" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35519 } }
I20260927 21:31:46.637214 19238 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ebb7028eec54c56a8659f91e4a8f644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:31:46.637322 19238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ebb7028eec54c56a8659f91e4a8f644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebb7028eec54c56a8659f91e4a8f644, State: Initialized, Role: FOLLOWER
I20260927 21:31:46.637998 19238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ebb7028eec54c56a8659f91e4a8f644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ebb7028eec54c56a8659f91e4a8f644" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35519 } }
I20260927 21:31:46.638271 19238 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ebb7028eec54c56a8659f91e4a8f644 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:31:46.638408 19238 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ebb7028eec54c56a8659f91e4a8f644 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:31:46.638592 19238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ebb7028eec54c56a8659f91e4a8f644 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:31:46.640749 19238 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ebb7028eec54c56a8659f91e4a8f644 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ebb7028eec54c56a8659f91e4a8f644" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35519 } }
I20260927 21:31:46.641279 19238 leader_election.cc:302] T 00000000000000000000000000000000 P 9ebb7028eec54c56a8659f91e4a8f644 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ebb7028eec54c56a8659f91e4a8f644; no voters:
I20260927 21:31:46.641881 19238 leader_election.cc:288] T 00000000000000000000000000000000 P 9ebb7028eec54c56a8659f91e4a8f644 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:31:46.642010 19243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ebb7028eec54c56a8659f91e4a8f644 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:31:46.642843 19243 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ebb7028eec54c56a8659f91e4a8f644 [term 1 LEADER]: Becoming Leader. State: Replica: 9ebb7028eec54c56a8659f91e4a8f644, State: Running, Role: LEADER
I20260927 21:31:46.643410 19243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ebb7028eec54c56a8659f91e4a8f644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ebb7028eec54c56a8659f91e4a8f644" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35519 } }
I20260927 21:31:46.644181 19238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ebb7028eec54c56a8659f91e4a8f644 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:31:46.646631 19245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ebb7028eec54c56a8659f91e4a8f644 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ebb7028eec54c56a8659f91e4a8f644. Latest consensus state: current_term: 1 leader_uuid: "9ebb7028eec54c56a8659f91e4a8f644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ebb7028eec54c56a8659f91e4a8f644" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35519 } } }
I20260927 21:31:46.647176 19245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ebb7028eec54c56a8659f91e4a8f644 [sys.catalog]: This master's current role is: LEADER
I20260927 21:31:46.646658 19244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ebb7028eec54c56a8659f91e4a8f644 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ebb7028eec54c56a8659f91e4a8f644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ebb7028eec54c56a8659f91e4a8f644" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35519 } } }
I20260927 21:31:46.647970 19244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ebb7028eec54c56a8659f91e4a8f644 [sys.catalog]: This master's current role is: LEADER
I20260927 21:31:46.650960 19252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:31:46.656284 19252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:31:46.665114 19252 catalog_manager.cc:1347] Generated new cluster ID: 9b9580e005c14e46b7df45b9d4d87711
I20260927 21:31:46.665243 19252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:31:46.684589 19252 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:31:46.686029 19252 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:31:46.693010 19252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ebb7028eec54c56a8659f91e4a8f644: Generated new TSK 0
I20260927 21:31:46.693924 19252 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:31:46.710886 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35519
--builtin_ntp_servers=127.0.97.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:31:47.043917 19262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:47.044293 19262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:47.044486 19262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:47.052075 19262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:47.052215 19262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:47.052402 19262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:31:47.063138 19262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:47.064436 19262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:47.066318 19262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:47.079864 19268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:47.079954 19267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:47.079984 19270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:47.080806 19262 server_base.cc:1034] running on GCE node
I20260927 21:31:47.081461 19262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:47.082465 19262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:47.083704 19262 hybrid_clock.cc:648] HybridClock initialized: now 1790544707083654 us; error 42 us; skew 500 ppm
I20260927 21:31:47.084108 19262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:31:47.086619 19262 webserver.cc:460] Webserver started at http://127.0.97.1:37153/ using document root <none> and password file <none>
I20260927 21:31:47.087371 19262 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:47.087488 19262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:47.087841 19262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:31:47.090870 19262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-0/data/instance:
uuid: "f4a28fdbe5c14adeb3328033620bcc45"
format_stamp: "Formatted at 2026-09-27 21:31:47 on dist-test-slave-bxm0"
server_key: "ea071566ace6e05fd84ccb3593915006"
server_key_iv: "b1c2e4ef968d58931ffe2db7097a17d3"
server_key_version: "encryptionkey@0"
I20260927 21:31:47.091676 19262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f4a28fdbe5c14adeb3328033620bcc45"
format_stamp: "Formatted at 2026-09-27 21:31:47 on dist-test-slave-bxm0"
server_key: "ea071566ace6e05fd84ccb3593915006"
server_key_iv: "b1c2e4ef968d58931ffe2db7097a17d3"
server_key_version: "encryptionkey@0"
I20260927 21:31:47.096884 19262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 21:31:47.100591 19276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:47.102097 19262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:31:47.102298 19262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "f4a28fdbe5c14adeb3328033620bcc45"
format_stamp: "Formatted at 2026-09-27 21:31:47 on dist-test-slave-bxm0"
server_key: "ea071566ace6e05fd84ccb3593915006"
server_key_iv: "b1c2e4ef968d58931ffe2db7097a17d3"
server_key_version: "encryptionkey@0"
I20260927 21:31:47.102533 19262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:47.140422 19262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:47.141323 19262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:47.141667 19262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:47.142666 19262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:31:47.144224 19262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:31:47.144352 19262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:47.144474 19262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:31:47.144568 19262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:47.178076 19262 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35555
I20260927 21:31:47.178166 19388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35555 every 8 connection(s)
I20260927 21:31:47.179433 19262 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:31:47.187157 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 19262
I20260927 21:31:47.187338 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-0/wal/instance
I20260927 21:31:47.187873 388 external_mini_cluster.cc:1428] Setting key c02d3f4c86ccca75f266e11fb9bb7a2c
I20260927 21:31:47.190974 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35519
--builtin_ntp_servers=127.0.97.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:31:47.193455 19391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:47.182205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48601 (local address 127.0.97.62:35519)
0927 21:31:47.182667 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:47.182673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:47.183864 (+ 1191us) server_negotiation.cc:407] Connection header received
0927 21:31:47.185564 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:47.185568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:47.185695 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:47.185794 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:47.186817 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:47.187999 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:47.190129 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:47.190783 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:47.190921 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:47.192505 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:47.192513 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:47.192521 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:31:47.192560 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:47.192633 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:31:47.192653 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:47.192736 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:47.193008 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 21:31:47.193126 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":217,"threads_started":1}
I20260927 21:31:47.194893 19392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:47.182809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35519 (local address 127.0.97.1:48601)
0927 21:31:47.183695 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:47.183731 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:47.185323 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:47.185989 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:47.185997 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:47.186083 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:31:47.186648 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:47.186662 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:47.188158 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:47.188162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:47.189991 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:47.190003 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:47.190981 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:47.190985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:47.191289 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:47.192079 (+ 790us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:31:47.192086 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:31:47.192143 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:31:47.192269 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:31:47.192771 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:31:47.192785 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:31:47.192797 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 21:31:47.192994 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 21:31:47.193446 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":286,"threads_started":1}
I20260927 21:31:47.196647 19389 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35519
I20260927 21:31:47.196987 19389 heartbeater.cc:464] Registering TS with master...
I20260927 21:31:47.197980 19389 heartbeater.cc:511] Master 127.0.97.62:35519 requested a full tablet report, sending...
I20260927 21:31:47.200574 19203 ts_manager.cc:194] Registered new tserver with Master: f4a28fdbe5c14adeb3328033620bcc45 (127.0.97.1:35555)
I20260927 21:31:47.202126 19203 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:48601
W20260927 21:31:47.523692 19393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:47.524032 19393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:47.524199 19393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:47.531515 19393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:47.531688 19393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:47.531884 19393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:31:47.541164 19393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:47.542522 19393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:47.544404 19393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:47.557904 19398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:47.557904 19399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:47.557945 19401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:47.558624 19393 server_base.cc:1034] running on GCE node
I20260927 21:31:47.559278 19393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:47.560309 19393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:47.561545 19393 hybrid_clock.cc:648] HybridClock initialized: now 1790544707561501 us; error 57 us; skew 500 ppm
I20260927 21:31:47.561916 19393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:31:47.564421 19393 webserver.cc:460] Webserver started at http://127.0.97.2:33465/ using document root <none> and password file <none>
I20260927 21:31:47.565153 19393 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:47.565291 19393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:47.565693 19393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:31:47.568792 19393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-1/data/instance:
uuid: "d061e8e1e40343d9a8a639cb3729cbba"
format_stamp: "Formatted at 2026-09-27 21:31:47 on dist-test-slave-bxm0"
server_key: "a008e43d8a8e75e3a990377a20018b95"
server_key_iv: "d37fadd3fe832eea2381867889bd4d82"
server_key_version: "encryptionkey@0"
I20260927 21:31:47.569667 19393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d061e8e1e40343d9a8a639cb3729cbba"
format_stamp: "Formatted at 2026-09-27 21:31:47 on dist-test-slave-bxm0"
server_key: "a008e43d8a8e75e3a990377a20018b95"
server_key_iv: "d37fadd3fe832eea2381867889bd4d82"
server_key_version: "encryptionkey@0"
I20260927 21:31:47.574944 19393 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 21:31:47.578790 19407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:47.580199 19393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:31:47.580394 19393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "d061e8e1e40343d9a8a639cb3729cbba"
format_stamp: "Formatted at 2026-09-27 21:31:47 on dist-test-slave-bxm0"
server_key: "a008e43d8a8e75e3a990377a20018b95"
server_key_iv: "d37fadd3fe832eea2381867889bd4d82"
server_key_version: "encryptionkey@0"
I20260927 21:31:47.580648 19393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:47.598052 19393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:47.598935 19393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:47.599225 19393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:47.600270 19393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:31:47.601799 19393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:31:47.601935 19393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:47.602061 19393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:31:47.602150 19393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:47.642786 19393 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36463
I20260927 21:31:47.642846 19519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36463 every 8 connection(s)
I20260927 21:31:47.644152 19393 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:31:47.649925 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 19393
I20260927 21:31:47.650110 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-1/wal/instance
I20260927 21:31:47.650545 388 external_mini_cluster.cc:1428] Setting key 8a22ce17a0a45fc983ba1d500a2ba1bf
I20260927 21:31:47.653290 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35519
--builtin_ntp_servers=127.0.97.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:31:47.657495 19391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:47.647030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58237 (local address 127.0.97.62:35519)
0927 21:31:47.647211 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:47.647216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:47.648916 (+ 1700us) server_negotiation.cc:407] Connection header received
0927 21:31:47.650940 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:47.650948 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:47.651038 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:47.651185 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:47.652315 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:47.653012 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:47.654748 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:47.655152 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:47.655287 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:47.656693 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:47.656698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:47.656701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:31:47.656721 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:47.656753 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:31:47.656765 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:47.656863 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:47.657090 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 21:31:47.657191 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 21:31:47.658661 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:47.647572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35519 (local address 127.0.97.2:58237)
0927 21:31:47.648743 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:47.648788 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:47.650612 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:47.651442 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:47.651451 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:47.651546 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:31:47.652122 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:47.652139 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:47.653164 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:47.653170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:47.654603 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:47.654615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:47.655403 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:47.655413 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:47.655636 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:47.656373 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:31:47.656375 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:31:47.656416 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:31:47.656500 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:31:47.656862 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:31:47.656874 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:31:47.656885 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 21:31:47.657082 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 21:31:47.657507 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":398,"threads_started":1}
I20260927 21:31:47.660097 19520 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35519
I20260927 21:31:47.660425 19520 heartbeater.cc:464] Registering TS with master...
I20260927 21:31:47.661245 19520 heartbeater.cc:511] Master 127.0.97.62:35519 requested a full tablet report, sending...
I20260927 21:31:47.662832 19203 ts_manager.cc:194] Registered new tserver with Master: d061e8e1e40343d9a8a639cb3729cbba (127.0.97.2:36463)
I20260927 21:31:47.663790 19203 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:58237
W20260927 21:31:47.983896 19523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:47.984257 19523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:47.984418 19523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:47.992123 19523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:47.992273 19523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:47.992461 19523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:31:48.001884 19523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:48.003239 19523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:48.005069 19523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:48.021314 19528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:48.021354 19531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:48.021400 19529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:48.022420 19523 server_base.cc:1034] running on GCE node
I20260927 21:31:48.023118 19523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:48.024134 19523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:48.025394 19523 hybrid_clock.cc:648] HybridClock initialized: now 1790544708025329 us; error 49 us; skew 500 ppm
I20260927 21:31:48.025810 19523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:31:48.028204 19523 webserver.cc:460] Webserver started at http://127.0.97.3:42443/ using document root <none> and password file <none>
I20260927 21:31:48.028941 19523 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:48.029078 19523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:48.029409 19523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:31:48.032418 19523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-2/data/instance:
uuid: "7f905caab5f94567b9c17b1a0cac12f0"
format_stamp: "Formatted at 2026-09-27 21:31:48 on dist-test-slave-bxm0"
server_key: "4002cf3b5457eb33562cb3c1bf5bc22a"
server_key_iv: "6b0b551f6ad0e86544a0d14aa0f3a847"
server_key_version: "encryptionkey@0"
I20260927 21:31:48.033246 19523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7f905caab5f94567b9c17b1a0cac12f0"
format_stamp: "Formatted at 2026-09-27 21:31:48 on dist-test-slave-bxm0"
server_key: "4002cf3b5457eb33562cb3c1bf5bc22a"
server_key_iv: "6b0b551f6ad0e86544a0d14aa0f3a847"
server_key_version: "encryptionkey@0"
I20260927 21:31:48.038543 19523 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 21:31:48.041985 19537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:48.043321 19523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 21:31:48.043507 19523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "7f905caab5f94567b9c17b1a0cac12f0"
format_stamp: "Formatted at 2026-09-27 21:31:48 on dist-test-slave-bxm0"
server_key: "4002cf3b5457eb33562cb3c1bf5bc22a"
server_key_iv: "6b0b551f6ad0e86544a0d14aa0f3a847"
server_key_version: "encryptionkey@0"
I20260927 21:31:48.043735 19523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:48.072777 19523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:48.073686 19523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:48.073971 19523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:48.074960 19523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:31:48.076668 19523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:31:48.076797 19523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:48.076923 19523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:31:48.077013 19523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:48.110502 19523 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41259
I20260927 21:31:48.110551 19649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41259 every 8 connection(s)
I20260927 21:31:48.111877 19523 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:31:48.119278 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 19523
I20260927 21:31:48.119463 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-2/wal/instance
I20260927 21:31:48.120093 388 external_mini_cluster.cc:1428] Setting key 6a28e5117e7dc1197c0699eb9571e800
I20260927 21:31:48.126555 19391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:48.114504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35017 (local address 127.0.97.62:35519)
0927 21:31:48.114684 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:48.114689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:48.116008 (+ 1319us) server_negotiation.cc:407] Connection header received
0927 21:31:48.117867 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:48.117873 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:48.117969 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:48.118082 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:48.119573 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.120729 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.123096 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.123791 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.123918 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:48.125285 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:48.125290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:48.125294 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:31:48.125312 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:48.125342 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:31:48.125353 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:48.125495 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:48.125774 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 21:31:48.125958 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:31:48.127413 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:48.114980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35519 (local address 127.0.97.3:35017)
0927 21:31:48.115849 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:48.115890 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:48.117529 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:48.118393 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:48.118404 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:48.118525 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:31:48.119348 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:48.119376 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.120957 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.120964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:48.122924 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:48.122939 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.123998 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.124004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:48.124224 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:48.124979 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:31:48.124981 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:31:48.125023 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:31:48.125109 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:31:48.125484 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:31:48.125494 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:31:48.125502 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 21:31:48.125738 (+ 236us) client_negotiation.cc:240] Negotiation successful
0927 21:31:48.126187 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":277,"threads_started":1}
I20260927 21:31:48.128755 19650 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35519
I20260927 21:31:48.129024 19650 heartbeater.cc:464] Registering TS with master...
I20260927 21:31:48.129825 19650 heartbeater.cc:511] Master 127.0.97.62:35519 requested a full tablet report, sending...
I20260927 21:31:48.131176 19203 ts_manager.cc:194] Registered new tserver with Master: 7f905caab5f94567b9c17b1a0cac12f0 (127.0.97.3:41259)
I20260927 21:31:48.131939 19203 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:35017
I20260927 21:31:48.135638 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:31:48.146312 19391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:48.139793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40106 (local address 127.0.97.62:35519)
0927 21:31:48.139991 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:48.139996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:48.140441 (+ 445us) server_negotiation.cc:407] Connection header received
0927 21:31:48.140616 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:48.140624 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:48.140733 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:48.140864 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:48.141550 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.142458 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.143904 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.144319 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.144415 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:48.145613 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:48.145622 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:48.145658 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:31:48.145680 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:48.145721 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:31:48.145734 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:48.145842 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:48.146038 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 21:31:48.146084 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 21:31:48.150517 19203 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:31:48.157337 19203 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:31:48.196404 19664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:48.170214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43454 (local address 127.0.97.1:35555)
0927 21:31:48.185744 (+ 15530us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:48.185759 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:48.185786 (+ 27us) server_negotiation.cc:407] Connection header received
0927 21:31:48.185873 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:48.185879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:48.186072 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:48.186284 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:48.187054 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.189321 (+ 2267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.193050 (+ 3729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.195134 (+ 2084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.195280 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:48.195599 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:48.195853 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 21:31:48.196031 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15272,"thread_start_us":15147,"threads_started":1}
I20260927 21:31:48.197212 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:48.169961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35555 (local address 127.0.0.1:43454)
0927 21:31:48.182787 (+ 12826us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:48.182868 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:48.183152 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:48.186298 (+ 3146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:48.186306 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:48.186339 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:48.186806 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:48.186827 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.189557 (+ 2730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.189566 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:48.192689 (+ 3123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:48.192715 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.195336 (+ 2621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.195342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:48.195558 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:48.195637 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 21:31:48.195791 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 21:31:48.195998 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12395,"thread_start_us":14163,"threads_started":1}
I20260927 21:31:48.201596 19324 tablet_service.cc:1467] Processing CreateTablet for tablet f122d330ba5c4472b7eaefcc9c7e3bee (DEFAULT_TABLE table=test-table [id=503318bbfe064d89b668efff44fcfe56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:31:48.202507 19663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:48.170131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41259 (local address 127.0.0.1:34040)
0927 21:31:48.183957 (+ 13826us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:48.183982 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:48.185028 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:48.194436 (+ 9408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:48.194441 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:48.194470 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:48.194861 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:48.194875 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.196914 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.196919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:48.199516 (+ 2597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:48.199526 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.201463 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.201476 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:48.201809 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:48.201844 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 21:31:48.201935 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 21:31:48.202070 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13642,"mutex_wait_us":53,"thread_start_us":16402,"threads_started":1}
I20260927 21:31:48.202534 19666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:48.170718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34040 (local address 127.0.97.3:41259)
0927 21:31:48.193792 (+ 23074us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:48.193802 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:48.193835 (+ 33us) server_negotiation.cc:407] Connection header received
0927 21:31:48.193934 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:48.193941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:48.194197 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:48.194416 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:48.195063 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.196591 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.199715 (+ 3124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.200919 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.201054 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:48.201321 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:48.201986 (+ 665us) server_negotiation.cc:299] Negotiation successful
0927 21:31:48.202166 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22860,"thread_start_us":22680,"threads_started":1}
I20260927 21:31:48.202723 19667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:48.171103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35764 (local address 127.0.97.2:36463)
0927 21:31:48.192369 (+ 21266us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:48.192378 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:48.192765 (+ 387us) server_negotiation.cc:407] Connection header received
0927 21:31:48.193040 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:48.193061 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:48.193455 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:48.193701 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:48.194697 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.196244 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.199385 (+ 3141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.200836 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.200979 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:48.201225 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:48.202199 (+ 974us) server_negotiation.cc:299] Negotiation successful
0927 21:31:48.202375 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21085,"thread_start_us":20962,"threads_started":1}
I20260927 21:31:48.203267 19665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:48.170506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36463 (local address 127.0.0.1:35764)
0927 21:31:48.192622 (+ 22116us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:48.192640 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:48.192825 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:48.193953 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:48.193958 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:48.193985 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:48.194443 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:48.194458 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.196407 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.196412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:48.199141 (+ 2729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:48.199158 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.201369 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.201377 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:48.201711 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:48.201782 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 21:31:48.201877 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 21:31:48.202020 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21995,"mutex_wait_us":506,"thread_start_us":22303,"threads_started":1}
I20260927 21:31:48.203585 19324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f122d330ba5c4472b7eaefcc9c7e3bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:31:48.206133 19389 heartbeater.cc:503] Master 127.0.97.62:35519 was elected leader, sending a full tablet report...
I20260927 21:31:48.206799 19455 tablet_service.cc:1467] Processing CreateTablet for tablet f122d330ba5c4472b7eaefcc9c7e3bee (DEFAULT_TABLE table=test-table [id=503318bbfe064d89b668efff44fcfe56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:31:48.206832 19585 tablet_service.cc:1467] Processing CreateTablet for tablet f122d330ba5c4472b7eaefcc9c7e3bee (DEFAULT_TABLE table=test-table [id=503318bbfe064d89b668efff44fcfe56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:31:48.208086 19455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f122d330ba5c4472b7eaefcc9c7e3bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:31:48.208283 19585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f122d330ba5c4472b7eaefcc9c7e3bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:31:48.219238 19668 tablet_bootstrap.cc:492] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45: Bootstrap starting.
I20260927 21:31:48.222513 19669 tablet_bootstrap.cc:492] T f122d330ba5c4472b7eaefcc9c7e3bee P d061e8e1e40343d9a8a639cb3729cbba: Bootstrap starting.
I20260927 21:31:48.223735 19668 tablet_bootstrap.cc:654] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45: Neither blocks nor log segments found. Creating new log.
I20260927 21:31:48.223963 19670 tablet_bootstrap.cc:492] T f122d330ba5c4472b7eaefcc9c7e3bee P 7f905caab5f94567b9c17b1a0cac12f0: Bootstrap starting.
I20260927 21:31:48.225834 19668 log.cc:824] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45: Log is configured to *not* fsync() on all Append() calls
I20260927 21:31:48.226524 19669 tablet_bootstrap.cc:654] T f122d330ba5c4472b7eaefcc9c7e3bee P d061e8e1e40343d9a8a639cb3729cbba: Neither blocks nor log segments found. Creating new log.
I20260927 21:31:48.227801 19669 log.cc:824] T f122d330ba5c4472b7eaefcc9c7e3bee P d061e8e1e40343d9a8a639cb3729cbba: Log is configured to *not* fsync() on all Append() calls
I20260927 21:31:48.227885 19670 tablet_bootstrap.cc:654] T f122d330ba5c4472b7eaefcc9c7e3bee P 7f905caab5f94567b9c17b1a0cac12f0: Neither blocks nor log segments found. Creating new log.
I20260927 21:31:48.229101 19670 log.cc:824] T f122d330ba5c4472b7eaefcc9c7e3bee P 7f905caab5f94567b9c17b1a0cac12f0: Log is configured to *not* fsync() on all Append() calls
I20260927 21:31:48.229743 19668 tablet_bootstrap.cc:492] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45: No bootstrap required, opened a new log
I20260927 21:31:48.230028 19668 ts_tablet_manager.cc:1397] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260927 21:31:48.230799 19669 tablet_bootstrap.cc:492] T f122d330ba5c4472b7eaefcc9c7e3bee P d061e8e1e40343d9a8a639cb3729cbba: No bootstrap required, opened a new log
I20260927 21:31:48.231078 19669 ts_tablet_manager.cc:1397] T f122d330ba5c4472b7eaefcc9c7e3bee P d061e8e1e40343d9a8a639cb3729cbba: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260927 21:31:48.232312 19670 tablet_bootstrap.cc:492] T f122d330ba5c4472b7eaefcc9c7e3bee P 7f905caab5f94567b9c17b1a0cac12f0: No bootstrap required, opened a new log
I20260927 21:31:48.232574 19670 ts_tablet_manager.cc:1397] T f122d330ba5c4472b7eaefcc9c7e3bee P 7f905caab5f94567b9c17b1a0cac12f0: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260927 21:31:48.239055 19669 raft_consensus.cc:348] T f122d330ba5c4472b7eaefcc9c7e3bee P d061e8e1e40343d9a8a639cb3729cbba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a28fdbe5c14adeb3328033620bcc45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35555 } } peers { permanent_uuid: "7f905caab5f94567b9c17b1a0cac12f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } peers { permanent_uuid: "d061e8e1e40343d9a8a639cb3729cbba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36463 } }
I20260927 21:31:48.239478 19669 raft_consensus.cc:374] T f122d330ba5c4472b7eaefcc9c7e3bee P d061e8e1e40343d9a8a639cb3729cbba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:31:48.239576 19669 raft_consensus.cc:729] T f122d330ba5c4472b7eaefcc9c7e3bee P d061e8e1e40343d9a8a639cb3729cbba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d061e8e1e40343d9a8a639cb3729cbba, State: Initialized, Role: FOLLOWER
I20260927 21:31:48.240231 19669 consensus_queue.cc:260] T f122d330ba5c4472b7eaefcc9c7e3bee P d061e8e1e40343d9a8a639cb3729cbba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a28fdbe5c14adeb3328033620bcc45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35555 } } peers { permanent_uuid: "7f905caab5f94567b9c17b1a0cac12f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } peers { permanent_uuid: "d061e8e1e40343d9a8a639cb3729cbba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36463 } }
I20260927 21:31:48.240953 19668 raft_consensus.cc:348] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a28fdbe5c14adeb3328033620bcc45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35555 } } peers { permanent_uuid: "7f905caab5f94567b9c17b1a0cac12f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } peers { permanent_uuid: "d061e8e1e40343d9a8a639cb3729cbba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36463 } }
I20260927 21:31:48.241390 19520 heartbeater.cc:503] Master 127.0.97.62:35519 was elected leader, sending a full tablet report...
I20260927 21:31:48.241420 19668 raft_consensus.cc:374] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:31:48.241518 19668 raft_consensus.cc:729] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4a28fdbe5c14adeb3328033620bcc45, State: Initialized, Role: FOLLOWER
I20260927 21:31:48.242130 19669 ts_tablet_manager.cc:1428] T f122d330ba5c4472b7eaefcc9c7e3bee P d061e8e1e40343d9a8a639cb3729cbba: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260927 21:31:48.242144 19668 consensus_queue.cc:260] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a28fdbe5c14adeb3328033620bcc45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35555 } } peers { permanent_uuid: "7f905caab5f94567b9c17b1a0cac12f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } peers { permanent_uuid: "d061e8e1e40343d9a8a639cb3729cbba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36463 } }
I20260927 21:31:48.244498 19670 raft_consensus.cc:348] T f122d330ba5c4472b7eaefcc9c7e3bee P 7f905caab5f94567b9c17b1a0cac12f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a28fdbe5c14adeb3328033620bcc45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35555 } } peers { permanent_uuid: "7f905caab5f94567b9c17b1a0cac12f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } peers { permanent_uuid: "d061e8e1e40343d9a8a639cb3729cbba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36463 } }
I20260927 21:31:48.245035 19670 raft_consensus.cc:374] T f122d330ba5c4472b7eaefcc9c7e3bee P 7f905caab5f94567b9c17b1a0cac12f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:31:48.245162 19670 raft_consensus.cc:729] T f122d330ba5c4472b7eaefcc9c7e3bee P 7f905caab5f94567b9c17b1a0cac12f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f905caab5f94567b9c17b1a0cac12f0, State: Initialized, Role: FOLLOWER
I20260927 21:31:48.245198 19668 ts_tablet_manager.cc:1428] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260927 21:31:48.245764 19670 consensus_queue.cc:260] T f122d330ba5c4472b7eaefcc9c7e3bee P 7f905caab5f94567b9c17b1a0cac12f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a28fdbe5c14adeb3328033620bcc45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35555 } } peers { permanent_uuid: "7f905caab5f94567b9c17b1a0cac12f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } peers { permanent_uuid: "d061e8e1e40343d9a8a639cb3729cbba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36463 } }
I20260927 21:31:48.246889 19650 heartbeater.cc:503] Master 127.0.97.62:35519 was elected leader, sending a full tablet report...
I20260927 21:31:48.247499 19670 ts_tablet_manager.cc:1428] T f122d330ba5c4472b7eaefcc9c7e3bee P 7f905caab5f94567b9c17b1a0cac12f0: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
I20260927 21:31:48.349726 19675 raft_consensus.cc:482] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:31:48.350114 19675 raft_consensus.cc:504] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a28fdbe5c14adeb3328033620bcc45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35555 } } peers { permanent_uuid: "7f905caab5f94567b9c17b1a0cac12f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } peers { permanent_uuid: "d061e8e1e40343d9a8a639cb3729cbba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36463 } }
I20260927 21:31:48.351980 19675 leader_election.cc:288] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f905caab5f94567b9c17b1a0cac12f0 (127.0.97.3:41259), d061e8e1e40343d9a8a639cb3729cbba (127.0.97.2:36463)
I20260927 21:31:48.361038 19667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:48.352390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60955 (local address 127.0.97.2:36463)
0927 21:31:48.352568 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:48.352573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:48.353288 (+ 715us) server_negotiation.cc:407] Connection header received
0927 21:31:48.353493 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:48.353498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:48.353587 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:48.353715 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:48.354428 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.355502 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.358567 (+ 3065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.359784 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.359932 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:48.360027 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:48.360656 (+ 629us) server_negotiation.cc:299] Negotiation successful
0927 21:31:48.360786 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 21:31:48.361038 19666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:48.352375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38437 (local address 127.0.97.3:41259)
0927 21:31:48.352563 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:48.352569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:48.353305 (+ 736us) server_negotiation.cc:407] Connection header received
0927 21:31:48.353507 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:48.353515 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:48.353587 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:48.353715 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:48.354428 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.355505 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.358569 (+ 3064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.359844 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.359954 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:48.360040 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:48.360660 (+ 620us) server_negotiation.cc:299] Negotiation successful
0927 21:31:48.360785 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:31:48.361224 19677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:48.352358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41259 (local address 127.0.97.1:38437)
0927 21:31:48.353217 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:48.353232 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:48.353385 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:48.353730 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:48.353736 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:48.353823 (+ 87us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:48.354264 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:48.354272 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.355652 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.355655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:48.358344 (+ 2689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:48.358360 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.360023 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.360028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:48.360371 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:48.360504 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 21:31:48.360598 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 21:31:48.360800 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":284,"spinlock_wait_cycles":22272,"thread_start_us":407,"threads_started":1}
I20260927 21:31:48.361714 19678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:48.352259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36463 (local address 127.0.97.1:60955)
0927 21:31:48.353156 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:48.353176 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:48.353338 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:48.353730 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:48.353736 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:48.353764 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:48.354242 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:48.354260 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.355650 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.355655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:48.358344 (+ 2689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:48.358360 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.360010 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.360017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:48.360371 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:48.360502 (+ 131us) client_negotiation.cc:769] Sending connection context
0927 21:31:48.360598 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 21:31:48.360800 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":381,"thread_start_us":536,"threads_started":1}
I20260927 21:31:48.362282 19605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f122d330ba5c4472b7eaefcc9c7e3bee" candidate_uuid: "f4a28fdbe5c14adeb3328033620bcc45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f905caab5f94567b9c17b1a0cac12f0" is_pre_election: true
I20260927 21:31:48.362833 19605 raft_consensus.cc:2454] T f122d330ba5c4472b7eaefcc9c7e3bee P 7f905caab5f94567b9c17b1a0cac12f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4a28fdbe5c14adeb3328033620bcc45 in term 0.
I20260927 21:31:48.363408 19475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f122d330ba5c4472b7eaefcc9c7e3bee" candidate_uuid: "f4a28fdbe5c14adeb3328033620bcc45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d061e8e1e40343d9a8a639cb3729cbba" is_pre_election: true
W20260927 21:31:48.363554 19651 tablet.cc:2369] T f122d330ba5c4472b7eaefcc9c7e3bee P 7f905caab5f94567b9c17b1a0cac12f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:31:48.363916 19475 raft_consensus.cc:2454] T f122d330ba5c4472b7eaefcc9c7e3bee P d061e8e1e40343d9a8a639cb3729cbba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4a28fdbe5c14adeb3328033620bcc45 in term 0.
I20260927 21:31:48.363950 19277 leader_election.cc:302] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f905caab5f94567b9c17b1a0cac12f0, f4a28fdbe5c14adeb3328033620bcc45; no voters:
I20260927 21:31:48.364364 19675 raft_consensus.cc:2781] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:31:48.364523 19675 raft_consensus.cc:482] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:31:48.364662 19675 raft_consensus.cc:3037] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:31:48.366976 19675 raft_consensus.cc:504] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a28fdbe5c14adeb3328033620bcc45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35555 } } peers { permanent_uuid: "7f905caab5f94567b9c17b1a0cac12f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } peers { permanent_uuid: "d061e8e1e40343d9a8a639cb3729cbba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36463 } }
I20260927 21:31:48.367566 19675 leader_election.cc:288] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45 [CANDIDATE]: Term 1 election: Requested vote from peers 7f905caab5f94567b9c17b1a0cac12f0 (127.0.97.3:41259), d061e8e1e40343d9a8a639cb3729cbba (127.0.97.2:36463)
I20260927 21:31:48.368036 19605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f122d330ba5c4472b7eaefcc9c7e3bee" candidate_uuid: "f4a28fdbe5c14adeb3328033620bcc45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f905caab5f94567b9c17b1a0cac12f0"
I20260927 21:31:48.368052 19475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f122d330ba5c4472b7eaefcc9c7e3bee" candidate_uuid: "f4a28fdbe5c14adeb3328033620bcc45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d061e8e1e40343d9a8a639cb3729cbba"
I20260927 21:31:48.368250 19605 raft_consensus.cc:3037] T f122d330ba5c4472b7eaefcc9c7e3bee P 7f905caab5f94567b9c17b1a0cac12f0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:31:48.368250 19475 raft_consensus.cc:3037] T f122d330ba5c4472b7eaefcc9c7e3bee P d061e8e1e40343d9a8a639cb3729cbba [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:31:48.371479 19475 raft_consensus.cc:2454] T f122d330ba5c4472b7eaefcc9c7e3bee P d061e8e1e40343d9a8a639cb3729cbba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4a28fdbe5c14adeb3328033620bcc45 in term 1.
I20260927 21:31:48.371479 19605 raft_consensus.cc:2454] T f122d330ba5c4472b7eaefcc9c7e3bee P 7f905caab5f94567b9c17b1a0cac12f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4a28fdbe5c14adeb3328033620bcc45 in term 1.
I20260927 21:31:48.371954 19279 leader_election.cc:302] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d061e8e1e40343d9a8a639cb3729cbba, f4a28fdbe5c14adeb3328033620bcc45; no voters:
I20260927 21:31:48.372304 19675 raft_consensus.cc:2781] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:31:48.372799 19675 raft_consensus.cc:686] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45 [term 1 LEADER]: Becoming Leader. State: Replica: f4a28fdbe5c14adeb3328033620bcc45, State: Running, Role: LEADER
I20260927 21:31:48.373270 19675 consensus_queue.cc:237] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a28fdbe5c14adeb3328033620bcc45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35555 } } peers { permanent_uuid: "7f905caab5f94567b9c17b1a0cac12f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } } peers { permanent_uuid: "d061e8e1e40343d9a8a639cb3729cbba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36463 } }
I20260927 21:31:48.377779 19202 catalog_manager.cc:5495] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4a28fdbe5c14adeb3328033620bcc45 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "f4a28fdbe5c14adeb3328033620bcc45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a28fdbe5c14adeb3328033620bcc45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f905caab5f94567b9c17b1a0cac12f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d061e8e1e40343d9a8a639cb3729cbba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36463 } health_report { overall_health: UNKNOWN } } }
W20260927 21:31:48.396438 19521 tablet.cc:2369] T f122d330ba5c4472b7eaefcc9c7e3bee P d061e8e1e40343d9a8a639cb3729cbba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:31:48.399947 19664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:48.394274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43470 (local address 127.0.97.1:35555)
0927 21:31:48.394483 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:48.394488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:48.394507 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:31:48.394691 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:48.394695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:48.394793 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:48.394928 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:31:48.395276 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.396015 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.397849 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.398275 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.398373 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:48.398831 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:31:48.399348 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:31:48.399475 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:48.399591 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 21:31:48.399661 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 21:31:48.408741 19605 raft_consensus.cc:1261] T f122d330ba5c4472b7eaefcc9c7e3bee P 7f905caab5f94567b9c17b1a0cac12f0 [term 1 FOLLOWER]: Refusing update from remote peer f4a28fdbe5c14adeb3328033620bcc45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:31:48.408740 19475 raft_consensus.cc:1261] T f122d330ba5c4472b7eaefcc9c7e3bee P d061e8e1e40343d9a8a639cb3729cbba [term 1 FOLLOWER]: Refusing update from remote peer f4a28fdbe5c14adeb3328033620bcc45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:31:48.409803 19679 consensus_queue.cc:1035] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f905caab5f94567b9c17b1a0cac12f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:31:48.410239 19675 consensus_queue.cc:1035] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d061e8e1e40343d9a8a639cb3729cbba" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:31:48.422919 19683 mvcc.cc:205] Tried to move back new op lower bound from 7334071125632303104 to 7334071125501165568. Current Snapshot: MvccSnapshot[applied={T|T < 7334071125632303104}]
I20260927 21:31:48.440526 19203 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:31:48.440814 19203 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:31:48.444473 19203 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ebb7028eec54c56a8659f91e4a8f644: Sending DeleteTablet for 3 replicas of tablet f122d330ba5c4472b7eaefcc9c7e3bee
I20260927 21:31:48.445761 19585 tablet_service.cc:1514] Processing DeleteTablet for tablet f122d330ba5c4472b7eaefcc9c7e3bee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:31:48 UTC) from {username='slave'} at 127.0.0.1:34040
I20260927 21:31:48.445796 19324 tablet_service.cc:1514] Processing DeleteTablet for tablet f122d330ba5c4472b7eaefcc9c7e3bee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:31:48 UTC) from {username='slave'} at 127.0.0.1:43454
I20260927 21:31:48.446419 19455 tablet_service.cc:1514] Processing DeleteTablet for tablet f122d330ba5c4472b7eaefcc9c7e3bee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:31:48 UTC) from {username='slave'} at 127.0.0.1:35764
I20260927 21:31:48.448251 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 19262
I20260927 21:31:48.461270 19692 tablet_replica.cc:330] T f122d330ba5c4472b7eaefcc9c7e3bee P 7f905caab5f94567b9c17b1a0cac12f0: stopping tablet replica
I20260927 21:31:48.461910 19692 raft_consensus.cc:2229] T f122d330ba5c4472b7eaefcc9c7e3bee P 7f905caab5f94567b9c17b1a0cac12f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:31:48.462530 19692 raft_consensus.cc:2258] T f122d330ba5c4472b7eaefcc9c7e3bee P 7f905caab5f94567b9c17b1a0cac12f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:31:48.463543 19693 tablet_replica.cc:330] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45: stopping tablet replica
I20260927 21:31:48.464151 19693 raft_consensus.cc:2229] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:31:48.465101 19693 raft_consensus.cc:2258] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:31:48.467716 19692 ts_tablet_manager.cc:1905] T f122d330ba5c4472b7eaefcc9c7e3bee P 7f905caab5f94567b9c17b1a0cac12f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:31:48.471776 19694 tablet_replica.cc:330] T f122d330ba5c4472b7eaefcc9c7e3bee P d061e8e1e40343d9a8a639cb3729cbba: stopping tablet replica
I20260927 21:31:48.472283 19694 raft_consensus.cc:2229] T f122d330ba5c4472b7eaefcc9c7e3bee P d061e8e1e40343d9a8a639cb3729cbba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:31:48.472817 19694 raft_consensus.cc:2258] T f122d330ba5c4472b7eaefcc9c7e3bee P d061e8e1e40343d9a8a639cb3729cbba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:31:48.473762 19693 ts_tablet_manager.cc:1905] T f122d330ba5c4472b7eaefcc9c7e3bee P f4a28fdbe5c14adeb3328033620bcc45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:31:48.475056 19692 ts_tablet_manager.cc:1918] T f122d330ba5c4472b7eaefcc9c7e3bee P 7f905caab5f94567b9c17b1a0cac12f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:31:48.475242 19692 log.cc:1196] T f122d330ba5c4472b7eaefcc9c7e3bee P 7f905caab5f94567b9c17b1a0cac12f0: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-2/wal/wals/f122d330ba5c4472b7eaefcc9c7e3bee
I20260927 21:31:48.475821 19692 ts_tablet_manager.cc:1939] T f122d330ba5c4472b7eaefcc9c7e3bee P 7f905caab5f94567b9c17b1a0cac12f0: Deleting consensus metadata
I20260927 21:31:48.477640 19664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:48.449890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43484 (local address 127.0.97.1:35555)
0927 21:31:48.450150 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:48.450163 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:48.466623 (+ 16460us) server_negotiation.cc:407] Connection header received
0927 21:31:48.467150 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:48.467155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:48.467261 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:48.467349 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:48.468055 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.469269 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.473229 (+ 3960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.474004 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.474104 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:48.475710 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:48.475728 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:48.475746 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:31:48.476636 (+ 890us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:48.476697 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:31:48.476724 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:48.476821 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:48.477100 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 21:31:48.477220 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 21:31:48.477842 19187 catalog_manager.cc:4841] TS 7f905caab5f94567b9c17b1a0cac12f0 (127.0.97.3:41259): tablet f122d330ba5c4472b7eaefcc9c7e3bee (table test-table [id=503318bbfe064d89b668efff44fcfe56]) successfully deleted
I20260927 21:31:48.478922 19384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:48.479966 19694 ts_tablet_manager.cc:1905] T f122d330ba5c4472b7eaefcc9c7e3bee P d061e8e1e40343d9a8a639cb3729cbba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:31:48.487622 19694 ts_tablet_manager.cc:1918] T f122d330ba5c4472b7eaefcc9c7e3bee P d061e8e1e40343d9a8a639cb3729cbba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:31:48.487816 19694 log.cc:1196] T f122d330ba5c4472b7eaefcc9c7e3bee P d061e8e1e40343d9a8a639cb3729cbba: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790544390748963-388-0/minicluster-data/ts-1/wal/wals/f122d330ba5c4472b7eaefcc9c7e3bee
I20260927 21:31:48.488427 19694 ts_tablet_manager.cc:1939] T f122d330ba5c4472b7eaefcc9c7e3bee P d061e8e1e40343d9a8a639cb3729cbba: Deleting consensus metadata
I20260927 21:31:48.490502 19189 catalog_manager.cc:4841] TS d061e8e1e40343d9a8a639cb3729cbba (127.0.97.2:36463): tablet f122d330ba5c4472b7eaefcc9c7e3bee (table test-table [id=503318bbfe064d89b668efff44fcfe56]) successfully deleted
I20260927 21:31:48.579480 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 19262
I20260927 21:31:48.597710 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 19393
W20260927 21:31:48.597772 19190 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:35555 (error 108)
I20260927 21:31:48.600056 19663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:48.599651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35555 (local address 127.0.0.1:43498)
0927 21:31:48.599815 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:48.599923 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 21:31:48.600327 19190 catalog_manager.cc:4584] TS f4a28fdbe5c14adeb3328033620bcc45 (127.0.97.1:35555): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f122d330ba5c4472b7eaefcc9c7e3bee: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35555: connect: Connection refused (error 111)
I20260927 21:31:48.604753 19667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:48.598502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35766 (local address 127.0.97.2:36463)
0927 21:31:48.598698 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:48.598704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:48.598808 (+ 104us) server_negotiation.cc:407] Connection header received
0927 21:31:48.599039 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:48.599045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:48.599148 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:48.599271 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:48.599651 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.600507 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.601869 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.602278 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.602414 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:48.603382 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:48.603399 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:48.603412 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:31:48.603974 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:48.604030 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:31:48.604049 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:48.604179 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:48.604305 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 21:31:48.604411 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 21:31:48.605443 19515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:48.654181 19663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:48.653494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35555 (local address 127.0.0.1:43514)
0927 21:31:48.653836 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:48.653950 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193}
I20260927 21:31:48.655544 19663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:48.655054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35555 (local address 127.0.0.1:43530)
0927 21:31:48.655293 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:48.655390 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260927 21:31:48.700994 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 19393
I20260927 21:31:48.716316 19663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:48.715839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35555 (local address 127.0.0.1:43538)
0927 21:31:48.716051 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:48.716156 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 21:31:48.717348 19663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:48.716986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35555 (local address 127.0.0.1:43554)
0927 21:31:48.717164 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:48.717233 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 21:31:48.718604 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 19523
I20260927 21:31:48.725838 19666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:48.719356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34056 (local address 127.0.97.3:41259)
0927 21:31:48.719562 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:48.719569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:48.719590 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:31:48.719788 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:48.719792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:48.719876 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:48.719989 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:48.720398 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.721178 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.722599 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:48.723093 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:48.723261 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:48.724362 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:48.724372 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:48.724387 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:31:48.725029 (+ 642us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:48.725094 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:31:48.725119 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:48.725232 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:48.725376 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 21:31:48.725469 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 21:31:48.726532 19645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:48.797883 19663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:48.797289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35555 (local address 127.0.0.1:43558)
0927 21:31:48.797535 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:48.797717 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260927 21:31:48.799010 19663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:48.798613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35555 (local address 127.0.0.1:43564)
0927 21:31:48.798821 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:48.798905 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260927 21:31:48.815712 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 19523
I20260927 21:31:48.832247 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 19172
I20260927 21:31:48.833302 19233 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:48.944476 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 19172
2026-09-27T21:31:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2860 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:31:49 dist-test-slave-bxm0 krb5kdc[19703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:31:49 dist-test-slave-bxm0 krb5kdc[19703](info): set up 2 sockets
Sep 27 21:31:49 dist-test-slave-bxm0 krb5kdc[19703](info): commencing operation
krb5kdc: starting...
W20260927 21:31:50.073920 388 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:31:50 dist-test-slave-bxm0 krb5kdc[19703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:31:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:31:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:31:50.333338 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33253
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:31:50.665297 19716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:50.665745 19716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:50.665921 19716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:50.673477 19716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:31:50.673600 19716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:50.673738 19716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:50.673844 19716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:31:50.673919 19716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:31:50.684197 19716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:50.685590 19716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:50.687057 19716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:50.699322 19721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:50.699337 19722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:50.699365 19724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:50.708330 19716 server_base.cc:1034] running on GCE node
I20260927 21:31:50.709295 19716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:50.710994 19716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:50.712225 19716 hybrid_clock.cc:648] HybridClock initialized: now 1790544710712175 us; error 41 us; skew 500 ppm
I20260927 21:31:50.712630 19716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:31:50 dist-test-slave-bxm0 krb5kdc[19703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:31:50.715948 19716 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:31:50.717657 19716 webserver.cc:460] Webserver started at http://127.0.97.62:38187/ using document root <none> and password file <none>
I20260927 21:31:50.718425 19716 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:50.718556 19716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:50.718895 19716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:31:50.721978 19716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "8a58cb6b48bf4bc3b5eebbc6d4b878b8"
format_stamp: "Formatted at 2026-09-27 21:31:50 on dist-test-slave-bxm0"
server_key: "8e5f7ad25105df8efea1277e10e5592b"
server_key_iv: "b8d192c89aa2ee133b308c45d272358b"
server_key_version: "encryptionkey@0"
I20260927 21:31:50.722731 19716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "8a58cb6b48bf4bc3b5eebbc6d4b878b8"
format_stamp: "Formatted at 2026-09-27 21:31:50 on dist-test-slave-bxm0"
server_key: "8e5f7ad25105df8efea1277e10e5592b"
server_key_iv: "b8d192c89aa2ee133b308c45d272358b"
server_key_version: "encryptionkey@0"
I20260927 21:31:50.728147 19716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:31:50.731855 19731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:50.733314 19716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:31:50.733526 19716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "8a58cb6b48bf4bc3b5eebbc6d4b878b8"
format_stamp: "Formatted at 2026-09-27 21:31:50 on dist-test-slave-bxm0"
server_key: "8e5f7ad25105df8efea1277e10e5592b"
server_key_iv: "b8d192c89aa2ee133b308c45d272358b"
server_key_version: "encryptionkey@0"
I20260927 21:31:50.733856 19716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:50.775038 19716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:50.779599 19716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:50.779927 19716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:50.799527 19716 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33253
I20260927 21:31:50.799626 19782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33253 every 8 connection(s)
I20260927 21:31:50.801061 19716 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:31:50.804908 19783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:31:50.808331 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 19716
I20260927 21:31:50.808538 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:31:50.808971 388 external_mini_cluster.cc:1428] Setting key a47550f87b2ff5a4d48b0d543acf7301
I20260927 21:31:50.817780 19783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a58cb6b48bf4bc3b5eebbc6d4b878b8: Bootstrap starting.
I20260927 21:31:50.821766 19783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a58cb6b48bf4bc3b5eebbc6d4b878b8: Neither blocks nor log segments found. Creating new log.
I20260927 21:31:50.823181 19783 log.cc:824] T 00000000000000000000000000000000 P 8a58cb6b48bf4bc3b5eebbc6d4b878b8: Log is configured to *not* fsync() on all Append() calls
Sep 27 21:31:50 dist-test-slave-bxm0 krb5kdc[19703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:31:50.826906 19783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a58cb6b48bf4bc3b5eebbc6d4b878b8: No bootstrap required, opened a new log
I20260927 21:31:50.834364 19786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:50.810884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56736 (local address 127.0.97.62:33253)
0927 21:31:50.811704 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:50.811715 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:50.811756 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:31:50.813467 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:50.813490 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:50.814002 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:50.814339 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:50.816110 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:50.817460 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:50.819334 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:50.820063 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:50.820201 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:50.825127 (+ 4926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:50.825139 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:50.825159 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:50.825207 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:50.829360 (+ 4153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:50.830244 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:50.830258 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:50.830267 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:50.830494 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:50.830903 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:50.830909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:50.830912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:50.831686 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:50.832012 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:50.832367 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 21:31:50.832859 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":398,"threads_started":1}
I20260927 21:31:50.837170 19783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a58cb6b48bf4bc3b5eebbc6d4b878b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a58cb6b48bf4bc3b5eebbc6d4b878b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33253 } }
I20260927 21:31:50.837656 19783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a58cb6b48bf4bc3b5eebbc6d4b878b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:31:50.837757 19783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a58cb6b48bf4bc3b5eebbc6d4b878b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a58cb6b48bf4bc3b5eebbc6d4b878b8, State: Initialized, Role: FOLLOWER
I20260927 21:31:50.838287 19783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a58cb6b48bf4bc3b5eebbc6d4b878b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a58cb6b48bf4bc3b5eebbc6d4b878b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33253 } }
I20260927 21:31:50.838564 19783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a58cb6b48bf4bc3b5eebbc6d4b878b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:31:50.838717 19783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a58cb6b48bf4bc3b5eebbc6d4b878b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:31:50.838907 19783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a58cb6b48bf4bc3b5eebbc6d4b878b8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:31:50.841149 19783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a58cb6b48bf4bc3b5eebbc6d4b878b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a58cb6b48bf4bc3b5eebbc6d4b878b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33253 } }
I20260927 21:31:50.841766 19783 leader_election.cc:302] T 00000000000000000000000000000000 P 8a58cb6b48bf4bc3b5eebbc6d4b878b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a58cb6b48bf4bc3b5eebbc6d4b878b8; no voters:
I20260927 21:31:50.842464 19783 leader_election.cc:288] T 00000000000000000000000000000000 P 8a58cb6b48bf4bc3b5eebbc6d4b878b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:31:50.842617 19788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a58cb6b48bf4bc3b5eebbc6d4b878b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:31:50.843307 19788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a58cb6b48bf4bc3b5eebbc6d4b878b8 [term 1 LEADER]: Becoming Leader. State: Replica: 8a58cb6b48bf4bc3b5eebbc6d4b878b8, State: Running, Role: LEADER
I20260927 21:31:50.843892 19788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a58cb6b48bf4bc3b5eebbc6d4b878b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a58cb6b48bf4bc3b5eebbc6d4b878b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33253 } }
I20260927 21:31:50.844635 19783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a58cb6b48bf4bc3b5eebbc6d4b878b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:31:50.846985 19789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a58cb6b48bf4bc3b5eebbc6d4b878b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a58cb6b48bf4bc3b5eebbc6d4b878b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a58cb6b48bf4bc3b5eebbc6d4b878b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33253 } } }
I20260927 21:31:50.847508 19789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a58cb6b48bf4bc3b5eebbc6d4b878b8 [sys.catalog]: This master's current role is: LEADER
I20260927 21:31:50.848519 19794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:31:50.848280 19790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a58cb6b48bf4bc3b5eebbc6d4b878b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a58cb6b48bf4bc3b5eebbc6d4b878b8. Latest consensus state: current_term: 1 leader_uuid: "8a58cb6b48bf4bc3b5eebbc6d4b878b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a58cb6b48bf4bc3b5eebbc6d4b878b8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33253 } } }
I20260927 21:31:50.850245 19790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a58cb6b48bf4bc3b5eebbc6d4b878b8 [sys.catalog]: This master's current role is: LEADER
I20260927 21:31:50.853225 19794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:31:50.863881 19794 catalog_manager.cc:1347] Generated new cluster ID: 7e2552251ba44cba808032b753500e30
I20260927 21:31:50.864041 19794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:31:50.889380 19794 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:31:50.891436 19794 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:31:50.901777 19794 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a58cb6b48bf4bc3b5eebbc6d4b878b8: Generated new TSK 0
I20260927 21:31:50.902855 19794 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 21:31:50.961499 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33253
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:31:51.312939 19809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:51.313246 19809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:51.313403 19809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:51.321058 19809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:51.321188 19809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:51.321396 19809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:31:51.331629 19809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:51.333047 19809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:51.334462 19809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:51.347886 19814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:51.348008 19817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:51.347846 19815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:51.349143 19809 server_base.cc:1034] running on GCE node
I20260927 21:31:51.349956 19809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:51.351027 19809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:51.352296 19809 hybrid_clock.cc:648] HybridClock initialized: now 1790544711352247 us; error 58 us; skew 500 ppm
I20260927 21:31:51.352658 19809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:31:51 dist-test-slave-bxm0 krb5kdc[19703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:31:51.356491 19809 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 21:31:51.358155 19809 webserver.cc:460] Webserver started at http://127.0.97.1:35119/ using document root <none> and password file <none>
I20260927 21:31:51.358865 19809 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:51.358976 19809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:51.359268 19809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:31:51.362483 19809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-0/data/instance:
uuid: "a344fd139d98495eb26a51392651f9c0"
format_stamp: "Formatted at 2026-09-27 21:31:51 on dist-test-slave-bxm0"
server_key: "6a869d59e3e6b173c3ba2d2fa2d094cf"
server_key_iv: "1436c37bb96a5eb16c89c2f7b4f9e573"
server_key_version: "encryptionkey@0"
I20260927 21:31:51.363329 19809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a344fd139d98495eb26a51392651f9c0"
format_stamp: "Formatted at 2026-09-27 21:31:51 on dist-test-slave-bxm0"
server_key: "6a869d59e3e6b173c3ba2d2fa2d094cf"
server_key_iv: "1436c37bb96a5eb16c89c2f7b4f9e573"
server_key_version: "encryptionkey@0"
I20260927 21:31:51.368968 19809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:31:51.373242 19824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:51.374749 19809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:31:51.374900 19809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "a344fd139d98495eb26a51392651f9c0"
format_stamp: "Formatted at 2026-09-27 21:31:51 on dist-test-slave-bxm0"
server_key: "6a869d59e3e6b173c3ba2d2fa2d094cf"
server_key_iv: "1436c37bb96a5eb16c89c2f7b4f9e573"
server_key_version: "encryptionkey@0"
I20260927 21:31:51.375087 19809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:51.396162 19809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:51.400828 19809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:51.401082 19809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:51.402177 19809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:31:51.403509 19809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:31:51.403581 19809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:51.403680 19809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:31:51.403723 19809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:51.437108 19809 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33159
I20260927 21:31:51.437158 19936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33159 every 8 connection(s)
I20260927 21:31:51.438548 19809 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:31:51.446879 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 19809
I20260927 21:31:51.447080 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-0/wal/instance
I20260927 21:31:51.447714 388 external_mini_cluster.cc:1428] Setting key 40acb773c9cc9b59e990070588fabee5
Sep 27 21:31:51 dist-test-slave-bxm0 krb5kdc[19703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:31:51.459708 19939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:51.441099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46537 (local address 127.0.97.62:33253)
0927 21:31:51.441570 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:51.441576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:51.442579 (+ 1003us) server_negotiation.cc:407] Connection header received
0927 21:31:51.444475 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:51.444480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:51.444594 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:51.444724 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:51.446358 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:51.447171 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:51.449307 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:51.449846 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:51.449994 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:51.453601 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:51.453647 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:51.453655 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:51.453704 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:51.456520 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:51.457275 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:51.457280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:51.457286 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:51.457412 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:51.457924 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:51.457930 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:51.457934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:51.458202 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:51.458371 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:51.459102 (+ 731us) server_negotiation.cc:299] Negotiation successful
0927 21:31:51.459342 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":225,"threads_started":1}
I20260927 21:31:51.460706 19940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:51.441701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33253 (local address 127.0.97.1:46537)
0927 21:31:51.442398 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:51.442446 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:51.444123 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:51.444958 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:51.444966 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:51.445498 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:31:51.446191 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:51.446209 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:51.447348 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:51.447355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:51.449165 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:51.449179 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:51.450106 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:51.450115 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:51.450388 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:51.451125 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:31:51.451152 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:31:51.453363 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:31:51.456669 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:31:51.456677 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:31:51.456696 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:31:51.457139 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:31:51.457534 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:31:51.457539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:31:51.457543 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:31:51.457803 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:31:51.458332 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:31:51.458341 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:31:51.458699 (+ 358us) client_negotiation.cc:769] Sending connection context
0927 21:31:51.458986 (+ 287us) client_negotiation.cc:240] Negotiation successful
0927 21:31:51.459536 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":268,"threads_started":1}
I20260927 21:31:51.462476 19937 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33253
I20260927 21:31:51.462792 19937 heartbeater.cc:464] Registering TS with master...
I20260927 21:31:51.463549 19937 heartbeater.cc:511] Master 127.0.97.62:33253 requested a full tablet report, sending...
I20260927 21:31:51.465931 19748 ts_manager.cc:194] Registered new tserver with Master: a344fd139d98495eb26a51392651f9c0 (127.0.97.1:33159)
I20260927 21:31:51.468159 19748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46537
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 21:31:51.505719 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33253
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:31:51.845336 19943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:51.845739 19943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:51.845927 19943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:51.853365 19943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:51.853520 19943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:51.853744 19943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:31:51.864235 19943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:51.865595 19943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:51.867019 19943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:51.879988 19948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:51.880079 19951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:51.880035 19949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:51.881086 19943 server_base.cc:1034] running on GCE node
I20260927 21:31:51.881804 19943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:51.882793 19943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:51.884065 19943 hybrid_clock.cc:648] HybridClock initialized: now 1790544711884003 us; error 48 us; skew 500 ppm
I20260927 21:31:51.884454 19943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:31:51 dist-test-slave-bxm0 krb5kdc[19703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:31:51.891878 19943 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 21:31:51.893344 19943 webserver.cc:460] Webserver started at http://127.0.97.2:41175/ using document root <none> and password file <none>
I20260927 21:31:51.894117 19943 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:51.894217 19943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:51.894469 19943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:31:51.897580 19943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-1/data/instance:
uuid: "6183a260ab2d4e05bef2b50019a07346"
format_stamp: "Formatted at 2026-09-27 21:31:51 on dist-test-slave-bxm0"
server_key: "18b993779a020d31ad4fb9414f6cd84c"
server_key_iv: "cdc4200b0eb64713729c505fbd5a3b05"
server_key_version: "encryptionkey@0"
I20260927 21:31:51.898432 19943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6183a260ab2d4e05bef2b50019a07346"
format_stamp: "Formatted at 2026-09-27 21:31:51 on dist-test-slave-bxm0"
server_key: "18b993779a020d31ad4fb9414f6cd84c"
server_key_iv: "cdc4200b0eb64713729c505fbd5a3b05"
server_key_version: "encryptionkey@0"
I20260927 21:31:51.903664 19943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:31:51.907159 19958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:51.908658 19943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:31:51.908835 19943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "6183a260ab2d4e05bef2b50019a07346"
format_stamp: "Formatted at 2026-09-27 21:31:51 on dist-test-slave-bxm0"
server_key: "18b993779a020d31ad4fb9414f6cd84c"
server_key_iv: "cdc4200b0eb64713729c505fbd5a3b05"
server_key_version: "encryptionkey@0"
I20260927 21:31:51.909020 19943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:51.932474 19943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:51.937067 19943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:51.937307 19943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:51.938429 19943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:31:51.939822 19943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:31:51.939900 19943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:51.939985 19943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:31:51.940027 19943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:51.973973 19943 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39677
I20260927 21:31:51.974066 20070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39677 every 8 connection(s)
I20260927 21:31:51.975281 19943 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:31:51.981383 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 19943
I20260927 21:31:51.981676 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-1/wal/instance
I20260927 21:31:51.982165 388 external_mini_cluster.cc:1428] Setting key 3293b95db028271b8765936b6546f266
Sep 27 21:31:51 dist-test-slave-bxm0 krb5kdc[19703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:31:51.998238 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:51.977989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51363 (local address 127.0.97.62:33253)
0927 21:31:51.978465 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:51.978469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:51.979385 (+ 916us) server_negotiation.cc:407] Connection header received
0927 21:31:51.981416 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:51.981422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:51.981515 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:51.981710 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:51.983312 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:51.984099 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:51.985668 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:51.986215 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:51.986359 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:51.990778 (+ 4419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:51.990787 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:51.990794 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:51.990848 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:51.994316 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:51.995471 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:51.995475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:51.995477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:51.995635 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:51.996194 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:51.996199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:51.996202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:51.996494 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:51.996614 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:51.997693 (+ 1079us) server_negotiation.cc:299] Negotiation successful
0927 21:31:51.997946 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":249,"threads_started":1}
I20260927 21:31:51.999717 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:51.978443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33253 (local address 127.0.97.2:51363)
0927 21:31:51.979215 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:51.979259 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:51.981133 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:51.981955 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:51.981964 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:51.982542 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:31:51.983125 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:51.983143 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:51.984250 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:51.984260 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:51.985499 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:51.985511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:51.986574 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:51.986585 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:51.986958 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:51.987713 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:31:51.987747 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:31:51.990507 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:31:51.994518 (+ 4011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:31:51.994532 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:31:51.994553 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:31:51.995316 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:31:51.995790 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:31:51.995798 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:31:51.995805 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:31:51.996084 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:31:51.996637 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:31:51.996648 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:31:51.997167 (+ 519us) client_negotiation.cc:769] Sending connection context
0927 21:31:51.997509 (+ 342us) client_negotiation.cc:240] Negotiation successful
0927 21:31:51.998167 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":278,"threads_started":1}
I20260927 21:31:52.001462 20071 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33253
I20260927 21:31:52.001832 20071 heartbeater.cc:464] Registering TS with master...
I20260927 21:31:52.002753 20071 heartbeater.cc:511] Master 127.0.97.62:33253 requested a full tablet report, sending...
I20260927 21:31:52.004504 19748 ts_manager.cc:194] Registered new tserver with Master: 6183a260ab2d4e05bef2b50019a07346 (127.0.97.2:39677)
I20260927 21:31:52.005376 19748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51363
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 21:31:52.045015 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33253
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:31:52.384187 20077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:52.384557 20077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:52.384732 20077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:52.392195 20077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:52.392375 20077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:52.392551 20077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:31:52.402618 20077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:52.403967 20077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:52.405328 20077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:52.419176 20083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:52.419271 20082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:52.419257 20085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:52.420361 20077 server_base.cc:1034] running on GCE node
I20260927 21:31:52.421051 20077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:52.422154 20077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:52.423384 20077 hybrid_clock.cc:648] HybridClock initialized: now 1790544712423335 us; error 49 us; skew 500 ppm
I20260927 21:31:52.423702 20077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:31:52 dist-test-slave-bxm0 krb5kdc[19703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:31:52.426836 20077 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 21:31:52.428375 20077 webserver.cc:460] Webserver started at http://127.0.97.3:41343/ using document root <none> and password file <none>
I20260927 21:31:52.429070 20077 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:52.429214 20077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:52.429533 20077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:31:52.432525 20077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-2/data/instance:
uuid: "b6879451b5714c539667c34a14edbb92"
format_stamp: "Formatted at 2026-09-27 21:31:52 on dist-test-slave-bxm0"
server_key: "fa32b60daabcd41f6cf9f6b190f9f9e9"
server_key_iv: "589d506fe0948ca865209bdccea9274d"
server_key_version: "encryptionkey@0"
I20260927 21:31:52.433314 20077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b6879451b5714c539667c34a14edbb92"
format_stamp: "Formatted at 2026-09-27 21:31:52 on dist-test-slave-bxm0"
server_key: "fa32b60daabcd41f6cf9f6b190f9f9e9"
server_key_iv: "589d506fe0948ca865209bdccea9274d"
server_key_version: "encryptionkey@0"
I20260927 21:31:52.438838 20077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 21:31:52.442507 20092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:52.443835 20077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:31:52.444005 20077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "b6879451b5714c539667c34a14edbb92"
format_stamp: "Formatted at 2026-09-27 21:31:52 on dist-test-slave-bxm0"
server_key: "fa32b60daabcd41f6cf9f6b190f9f9e9"
server_key_iv: "589d506fe0948ca865209bdccea9274d"
server_key_version: "encryptionkey@0"
I20260927 21:31:52.444223 20077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:52.464573 20077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:52.469035 20077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:52.469357 20077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:52.470485 20077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:31:52.471433 19937 heartbeater.cc:503] Master 127.0.97.62:33253 was elected leader, sending a full tablet report...
I20260927 21:31:52.472101 20077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:31:52.472204 20077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:52.472369 20077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:31:52.472443 20077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:52.506958 20077 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42927
I20260927 21:31:52.507001 20204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42927 every 8 connection(s)
I20260927 21:31:52.508304 20077 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:31:52.513517 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 20077
I20260927 21:31:52.513728 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-2/wal/instance
I20260927 21:31:52.514256 388 external_mini_cluster.cc:1428] Setting key d0189c278096fe3546d3dc9bbad3d3c3
Sep 27 21:31:52 dist-test-slave-bxm0 krb5kdc[19703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:31:52.530543 20207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:52.511047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57379 (local address 127.0.97.62:33253)
0927 21:31:52.511560 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:52.511565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:52.512452 (+ 887us) server_negotiation.cc:407] Connection header received
0927 21:31:52.515065 (+ 2613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:52.515069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:52.515141 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:52.515270 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:52.517218 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.518084 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.519854 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.520257 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.520367 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:52.524014 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:52.524019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:52.524022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:52.524057 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:52.526800 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:52.527578 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:52.527582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:52.527584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:52.527745 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:52.528263 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:52.528266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:52.528267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:52.528722 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:52.528879 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:52.529879 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0927 21:31:52.530184 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":280,"threads_started":1}
I20260927 21:31:52.531509 20208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:52.511533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33253 (local address 127.0.97.3:57379)
0927 21:31:52.512272 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:52.512326 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:52.514742 (+ 2416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:52.515521 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:52.515537 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:52.516363 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:31:52.517031 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:52.517060 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.518247 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.518252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:52.519639 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:52.519656 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.520502 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.520508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:52.520729 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:52.521458 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:31:52.521486 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:31:52.523789 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:31:52.526946 (+ 3157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:31:52.526967 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:31:52.526982 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:31:52.527430 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:31:52.527862 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:31:52.527869 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:31:52.527873 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:31:52.528111 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:31:52.528868 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:31:52.528876 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:31:52.529394 (+ 518us) client_negotiation.cc:769] Sending connection context
0927 21:31:52.529752 (+ 358us) client_negotiation.cc:240] Negotiation successful
0927 21:31:52.530265 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":300,"threads_started":1}
I20260927 21:31:52.532892 20205 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33253
I20260927 21:31:52.533198 20205 heartbeater.cc:464] Registering TS with master...
I20260927 21:31:52.533970 20205 heartbeater.cc:511] Master 127.0.97.62:33253 requested a full tablet report, sending...
I20260927 21:31:52.535583 19748 ts_manager.cc:194] Registered new tserver with Master: b6879451b5714c539667c34a14edbb92 (127.0.97.3:42927)
I20260927 21:31:52.536406 19748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57379
I20260927 21:31:52.542817 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:31:52.559964 20207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:52.547320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37466 (local address 127.0.97.62:33253)
0927 21:31:52.547531 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:52.547536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:52.548030 (+ 494us) server_negotiation.cc:407] Connection header received
0927 21:31:52.548192 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:52.548200 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:52.548286 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:52.548411 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:52.549901 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.550648 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.551889 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.552265 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.552363 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:52.554801 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:52.554806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:52.554809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:52.554841 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:52.557550 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:52.558280 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:52.558284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:52.558286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:52.558424 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:52.558797 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:52.558799 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:52.558801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:52.559105 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:52.559286 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:52.559563 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 21:31:52.559745 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 21:31:52.563881 19748 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:31:52.571280 19748 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:31:52.595057 20220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:52.585076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33159 (local address 127.0.0.1:37442)
0927 21:31:52.585598 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:52.585656 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:52.585764 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:52.587034 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:52.587038 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:52.587061 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:52.587492 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:52.587505 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.589092 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.589098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:52.591752 (+ 2654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:52.591760 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.594227 (+ 2467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.594235 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:52.594392 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:52.594436 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 21:31:52.594521 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 21:31:52.594635 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":31,"thread_start_us":265,"threads_started":1}
I20260927 21:31:52.595168 20219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:52.584050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47934 (local address 127.0.97.3:42927)
0927 21:31:52.585338 (+ 1288us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:52.585347 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:52.585379 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:31:52.585485 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:52.585493 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:52.585777 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:52.586802 (+ 1025us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:52.587625 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.589299 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.591915 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.593376 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.593513 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:52.593785 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:52.594662 (+ 877us) server_negotiation.cc:299] Negotiation successful
0927 21:31:52.594832 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1136,"thread_start_us":954,"threads_started":1}
I20260927 21:31:52.595232 20218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:52.583782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42927 (local address 127.0.0.1:47934)
0927 21:31:52.584665 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:52.584700 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:52.584877 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:52.586113 (+ 1236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:52.586120 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:52.586153 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:52.586586 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:52.586600 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.589432 (+ 2832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.589437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:52.591742 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:52.591758 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.593596 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.593605 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:52.593937 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:52.593995 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 21:31:52.594163 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 21:31:52.594402 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":80,"thread_start_us":517,"threads_started":1}
I20260927 21:31:52.595403 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:52.585675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39677 (local address 127.0.0.1:58498)
0927 21:31:52.586622 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:52.586642 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:52.586752 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:52.587824 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:52.587830 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:52.587860 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:52.588193 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:52.588205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.590048 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.590053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:52.592349 (+ 2296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:52.592363 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.593799 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.593805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:52.593977 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:52.594002 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 21:31:52.594149 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 21:31:52.594559 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":410,"thread_start_us":661,"threads_started":1}
I20260927 21:31:52.595129 20223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:52.586146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58498 (local address 127.0.97.2:39677)
0927 21:31:52.587237 (+ 1091us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:52.587246 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:52.587273 (+ 27us) server_negotiation.cc:407] Connection header received
0927 21:31:52.587362 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:52.587370 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:52.587587 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:52.588367 (+ 780us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:52.588588 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.589907 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.592537 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.593522 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.594342 (+ 820us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:52.594535 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:52.594609 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 21:31:52.594783 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":938,"thread_start_us":800,"threads_started":1}
I20260927 21:31:52.596151 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:52.585656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37442 (local address 127.0.97.1:33159)
0927 21:31:52.586350 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:52.586360 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:52.586388 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:31:52.586483 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:52.586494 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:52.586804 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:52.587714 (+ 910us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:52.587966 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.588961 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.592555 (+ 3594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.593986 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.594162 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:52.594389 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:52.595602 (+ 1213us) server_negotiation.cc:299] Negotiation successful
0927 21:31:52.595791 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":339,"threads_started":1}
I20260927 21:31:52.599327 20140 tablet_service.cc:1467] Processing CreateTablet for tablet 74f5fd23f6994c31948028c089aa17d6 (DEFAULT_TABLE table=test-table [id=a7c909d61ef9417eb97925839954b87e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:31:52.599534 19872 tablet_service.cc:1467] Processing CreateTablet for tablet 74f5fd23f6994c31948028c089aa17d6 (DEFAULT_TABLE table=test-table [id=a7c909d61ef9417eb97925839954b87e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:31:52.599536 20006 tablet_service.cc:1467] Processing CreateTablet for tablet 74f5fd23f6994c31948028c089aa17d6 (DEFAULT_TABLE table=test-table [id=a7c909d61ef9417eb97925839954b87e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:31:52.600883 20140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f5fd23f6994c31948028c089aa17d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:31:52.600957 20006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f5fd23f6994c31948028c089aa17d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:31:52.600957 19872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f5fd23f6994c31948028c089aa17d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:31:52.612707 20224 tablet_bootstrap.cc:492] T 74f5fd23f6994c31948028c089aa17d6 P b6879451b5714c539667c34a14edbb92: Bootstrap starting.
I20260927 21:31:52.616925 20226 tablet_bootstrap.cc:492] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346: Bootstrap starting.
I20260927 21:31:52.617132 20224 tablet_bootstrap.cc:654] T 74f5fd23f6994c31948028c089aa17d6 P b6879451b5714c539667c34a14edbb92: Neither blocks nor log segments found. Creating new log.
I20260927 21:31:52.618932 20224 log.cc:824] T 74f5fd23f6994c31948028c089aa17d6 P b6879451b5714c539667c34a14edbb92: Log is configured to *not* fsync() on all Append() calls
I20260927 21:31:52.620738 20226 tablet_bootstrap.cc:654] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346: Neither blocks nor log segments found. Creating new log.
I20260927 21:31:52.620422 20225 tablet_bootstrap.cc:492] T 74f5fd23f6994c31948028c089aa17d6 P a344fd139d98495eb26a51392651f9c0: Bootstrap starting.
I20260927 21:31:52.622145 20226 log.cc:824] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346: Log is configured to *not* fsync() on all Append() calls
I20260927 21:31:52.622478 20224 tablet_bootstrap.cc:492] T 74f5fd23f6994c31948028c089aa17d6 P b6879451b5714c539667c34a14edbb92: No bootstrap required, opened a new log
I20260927 21:31:52.622799 20224 ts_tablet_manager.cc:1397] T 74f5fd23f6994c31948028c089aa17d6 P b6879451b5714c539667c34a14edbb92: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260927 21:31:52.625453 20225 tablet_bootstrap.cc:654] T 74f5fd23f6994c31948028c089aa17d6 P a344fd139d98495eb26a51392651f9c0: Neither blocks nor log segments found. Creating new log.
I20260927 21:31:52.625478 20226 tablet_bootstrap.cc:492] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346: No bootstrap required, opened a new log
I20260927 21:31:52.625797 20226 ts_tablet_manager.cc:1397] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 21:31:52.626792 20225 log.cc:824] T 74f5fd23f6994c31948028c089aa17d6 P a344fd139d98495eb26a51392651f9c0: Log is configured to *not* fsync() on all Append() calls
I20260927 21:31:52.630049 20225 tablet_bootstrap.cc:492] T 74f5fd23f6994c31948028c089aa17d6 P a344fd139d98495eb26a51392651f9c0: No bootstrap required, opened a new log
I20260927 21:31:52.630335 20225 ts_tablet_manager.cc:1397] T 74f5fd23f6994c31948028c089aa17d6 P a344fd139d98495eb26a51392651f9c0: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 21:31:52.631259 20224 raft_consensus.cc:348] T 74f5fd23f6994c31948028c089aa17d6 P b6879451b5714c539667c34a14edbb92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6879451b5714c539667c34a14edbb92" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42927 } } peers { permanent_uuid: "a344fd139d98495eb26a51392651f9c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33159 } } peers { permanent_uuid: "6183a260ab2d4e05bef2b50019a07346" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39677 } }
I20260927 21:31:52.631654 20224 raft_consensus.cc:374] T 74f5fd23f6994c31948028c089aa17d6 P b6879451b5714c539667c34a14edbb92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:31:52.631753 20224 raft_consensus.cc:729] T 74f5fd23f6994c31948028c089aa17d6 P b6879451b5714c539667c34a14edbb92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6879451b5714c539667c34a14edbb92, State: Initialized, Role: FOLLOWER
I20260927 21:31:52.632378 20224 consensus_queue.cc:260] T 74f5fd23f6994c31948028c089aa17d6 P b6879451b5714c539667c34a14edbb92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6879451b5714c539667c34a14edbb92" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42927 } } peers { permanent_uuid: "a344fd139d98495eb26a51392651f9c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33159 } } peers { permanent_uuid: "6183a260ab2d4e05bef2b50019a07346" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39677 } }
I20260927 21:31:52.633491 20205 heartbeater.cc:503] Master 127.0.97.62:33253 was elected leader, sending a full tablet report...
I20260927 21:31:52.635747 20224 ts_tablet_manager.cc:1428] T 74f5fd23f6994c31948028c089aa17d6 P b6879451b5714c539667c34a14edbb92: Time spent starting tablet: real 0.013s user 0.008s sys 0.002s
I20260927 21:31:52.636279 20226 raft_consensus.cc:348] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6879451b5714c539667c34a14edbb92" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42927 } } peers { permanent_uuid: "a344fd139d98495eb26a51392651f9c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33159 } } peers { permanent_uuid: "6183a260ab2d4e05bef2b50019a07346" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39677 } }
I20260927 21:31:52.636719 20226 raft_consensus.cc:374] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:31:52.636818 20226 raft_consensus.cc:729] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6183a260ab2d4e05bef2b50019a07346, State: Initialized, Role: FOLLOWER
I20260927 21:31:52.637441 20226 consensus_queue.cc:260] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6879451b5714c539667c34a14edbb92" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42927 } } peers { permanent_uuid: "a344fd139d98495eb26a51392651f9c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33159 } } peers { permanent_uuid: "6183a260ab2d4e05bef2b50019a07346" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39677 } }
I20260927 21:31:52.638590 20071 heartbeater.cc:503] Master 127.0.97.62:33253 was elected leader, sending a full tablet report...
I20260927 21:31:52.639266 20226 ts_tablet_manager.cc:1428] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260927 21:31:52.640993 20225 raft_consensus.cc:348] T 74f5fd23f6994c31948028c089aa17d6 P a344fd139d98495eb26a51392651f9c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6879451b5714c539667c34a14edbb92" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42927 } } peers { permanent_uuid: "a344fd139d98495eb26a51392651f9c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33159 } } peers { permanent_uuid: "6183a260ab2d4e05bef2b50019a07346" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39677 } }
I20260927 21:31:52.641381 20225 raft_consensus.cc:374] T 74f5fd23f6994c31948028c089aa17d6 P a344fd139d98495eb26a51392651f9c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:31:52.641484 20225 raft_consensus.cc:729] T 74f5fd23f6994c31948028c089aa17d6 P a344fd139d98495eb26a51392651f9c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a344fd139d98495eb26a51392651f9c0, State: Initialized, Role: FOLLOWER
I20260927 21:31:52.642134 20225 consensus_queue.cc:260] T 74f5fd23f6994c31948028c089aa17d6 P a344fd139d98495eb26a51392651f9c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6879451b5714c539667c34a14edbb92" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42927 } } peers { permanent_uuid: "a344fd139d98495eb26a51392651f9c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33159 } } peers { permanent_uuid: "6183a260ab2d4e05bef2b50019a07346" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39677 } }
I20260927 21:31:52.643714 20225 ts_tablet_manager.cc:1428] T 74f5fd23f6994c31948028c089aa17d6 P a344fd139d98495eb26a51392651f9c0: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260927 21:31:52.661780 20231 raft_consensus.cc:482] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:31:52.662125 20231 raft_consensus.cc:504] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6879451b5714c539667c34a14edbb92" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42927 } } peers { permanent_uuid: "a344fd139d98495eb26a51392651f9c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33159 } } peers { permanent_uuid: "6183a260ab2d4e05bef2b50019a07346" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39677 } }
I20260927 21:31:52.663959 20231 leader_election.cc:288] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6879451b5714c539667c34a14edbb92 (127.0.97.3:42927), a344fd139d98495eb26a51392651f9c0 (127.0.97.1:33159)
I20260927 21:31:52.670573 20219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:52.664486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50893 (local address 127.0.97.3:42927)
0927 21:31:52.664683 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:52.664702 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:52.665097 (+ 395us) server_negotiation.cc:407] Connection header received
0927 21:31:52.665445 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:52.665451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:52.665542 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:52.665594 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:52.666280 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.667023 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.668835 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.669583 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.669796 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:52.669918 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:52.670189 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 21:31:52.670315 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 21:31:52.670739 20233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:52.664290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42927 (local address 127.0.97.2:50893)
0927 21:31:52.665000 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:52.665019 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:52.665236 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:52.665827 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:52.665830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:52.665852 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:52.666128 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:52.666135 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.667176 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.667183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:52.668667 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:52.668679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.669856 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.669863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:52.670027 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:52.670050 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 21:31:52.670177 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 21:31:52.670314 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":2,"thread_start_us":348,"threads_started":1}
I20260927 21:31:52.670972 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:52.665355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39345 (local address 127.0.97.1:33159)
0927 21:31:52.665560 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:52.665567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:52.665788 (+ 221us) server_negotiation.cc:407] Connection header received
0927 21:31:52.665983 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:52.665993 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:52.666051 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:52.666148 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:52.666683 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.667425 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.669141 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.669982 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.670154 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:52.670261 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:52.670578 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 21:31:52.670712 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 21:31:52.671242 20234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:52.665083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33159 (local address 127.0.97.2:39345)
0927 21:31:52.665686 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:52.665702 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:52.665841 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:52.666163 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:52.666167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:52.666198 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:52.666533 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:52.666542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.667571 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.667578 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:52.668979 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:52.668990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.670259 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.670265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:52.670422 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:52.670447 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 21:31:52.670553 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 21:31:52.670681 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":20,"thread_start_us":271,"threads_started":1}
I20260927 21:31:52.671866 20160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f5fd23f6994c31948028c089aa17d6" candidate_uuid: "6183a260ab2d4e05bef2b50019a07346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6879451b5714c539667c34a14edbb92" is_pre_election: true
I20260927 21:31:52.672259 19892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f5fd23f6994c31948028c089aa17d6" candidate_uuid: "6183a260ab2d4e05bef2b50019a07346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a344fd139d98495eb26a51392651f9c0" is_pre_election: true
I20260927 21:31:52.672487 20160 raft_consensus.cc:2454] T 74f5fd23f6994c31948028c089aa17d6 P b6879451b5714c539667c34a14edbb92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6183a260ab2d4e05bef2b50019a07346 in term 0.
I20260927 21:31:52.672812 19892 raft_consensus.cc:2454] T 74f5fd23f6994c31948028c089aa17d6 P a344fd139d98495eb26a51392651f9c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6183a260ab2d4e05bef2b50019a07346 in term 0.
I20260927 21:31:52.673262 19960 leader_election.cc:302] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6183a260ab2d4e05bef2b50019a07346, b6879451b5714c539667c34a14edbb92; no voters:
I20260927 21:31:52.673672 20231 raft_consensus.cc:2781] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:31:52.673827 20231 raft_consensus.cc:482] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:31:52.673938 20231 raft_consensus.cc:3037] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:31:52.676232 20231 raft_consensus.cc:504] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6879451b5714c539667c34a14edbb92" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42927 } } peers { permanent_uuid: "a344fd139d98495eb26a51392651f9c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33159 } } peers { permanent_uuid: "6183a260ab2d4e05bef2b50019a07346" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39677 } }
I20260927 21:31:52.676821 20231 leader_election.cc:288] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346 [CANDIDATE]: Term 1 election: Requested vote from peers b6879451b5714c539667c34a14edbb92 (127.0.97.3:42927), a344fd139d98495eb26a51392651f9c0 (127.0.97.1:33159)
I20260927 21:31:52.677222 20160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f5fd23f6994c31948028c089aa17d6" candidate_uuid: "6183a260ab2d4e05bef2b50019a07346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6879451b5714c539667c34a14edbb92"
I20260927 21:31:52.677311 19892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f5fd23f6994c31948028c089aa17d6" candidate_uuid: "6183a260ab2d4e05bef2b50019a07346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a344fd139d98495eb26a51392651f9c0"
I20260927 21:31:52.677433 20160 raft_consensus.cc:3037] T 74f5fd23f6994c31948028c089aa17d6 P b6879451b5714c539667c34a14edbb92 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:31:52.677546 19892 raft_consensus.cc:3037] T 74f5fd23f6994c31948028c089aa17d6 P a344fd139d98495eb26a51392651f9c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:31:52.680011 19892 raft_consensus.cc:2454] T 74f5fd23f6994c31948028c089aa17d6 P a344fd139d98495eb26a51392651f9c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6183a260ab2d4e05bef2b50019a07346 in term 1.
I20260927 21:31:52.679958 20160 raft_consensus.cc:2454] T 74f5fd23f6994c31948028c089aa17d6 P b6879451b5714c539667c34a14edbb92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6183a260ab2d4e05bef2b50019a07346 in term 1.
I20260927 21:31:52.680428 19960 leader_election.cc:302] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6183a260ab2d4e05bef2b50019a07346, a344fd139d98495eb26a51392651f9c0; no voters:
I20260927 21:31:52.680740 20231 raft_consensus.cc:2781] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:31:52.681196 20231 raft_consensus.cc:686] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346 [term 1 LEADER]: Becoming Leader. State: Replica: 6183a260ab2d4e05bef2b50019a07346, State: Running, Role: LEADER
I20260927 21:31:52.681833 20231 consensus_queue.cc:237] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6879451b5714c539667c34a14edbb92" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42927 } } peers { permanent_uuid: "a344fd139d98495eb26a51392651f9c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33159 } } peers { permanent_uuid: "6183a260ab2d4e05bef2b50019a07346" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39677 } }
I20260927 21:31:52.686847 19748 catalog_manager.cc:5495] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6183a260ab2d4e05bef2b50019a07346 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "6183a260ab2d4e05bef2b50019a07346" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6879451b5714c539667c34a14edbb92" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a344fd139d98495eb26a51392651f9c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6183a260ab2d4e05bef2b50019a07346" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39677 } health_report { overall_health: HEALTHY } } }
W20260927 21:31:52.690909 19938 tablet.cc:2369] T 74f5fd23f6994c31948028c089aa17d6 P a344fd139d98495eb26a51392651f9c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:31:52.721866 20223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:52.716348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58510 (local address 127.0.97.2:39677)
0927 21:31:52.716641 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:52.716650 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:52.716687 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:31:52.716867 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:52.716872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:52.716974 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:52.717143 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:31:52.717496 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.718279 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.719658 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.720151 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.720257 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:52.720754 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:31:52.721261 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:31:52.721374 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:52.721454 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 21:31:52.721522 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 21:31:52.727344 20072 tablet.cc:2369] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:31:52.730294 19892 raft_consensus.cc:1261] T 74f5fd23f6994c31948028c089aa17d6 P a344fd139d98495eb26a51392651f9c0 [term 1 FOLLOWER]: Refusing update from remote peer 6183a260ab2d4e05bef2b50019a07346: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:31:52.730352 20160 raft_consensus.cc:1261] T 74f5fd23f6994c31948028c089aa17d6 P b6879451b5714c539667c34a14edbb92 [term 1 FOLLOWER]: Refusing update from remote peer 6183a260ab2d4e05bef2b50019a07346: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:31:52.731376 20235 consensus_queue.cc:1035] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a344fd139d98495eb26a51392651f9c0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:31:52.731755 20231 consensus_queue.cc:1035] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6879451b5714c539667c34a14edbb92" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:31:52.757014 19747 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:31:52.757355 19747 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:31:52.761458 19747 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a58cb6b48bf4bc3b5eebbc6d4b878b8: Sending DeleteTablet for 3 replicas of tablet 74f5fd23f6994c31948028c089aa17d6
I20260927 21:31:52.762625 20140 tablet_service.cc:1514] Processing DeleteTablet for tablet 74f5fd23f6994c31948028c089aa17d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:31:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47934
I20260927 21:31:52.762871 19872 tablet_service.cc:1514] Processing DeleteTablet for tablet 74f5fd23f6994c31948028c089aa17d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:31:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37442
I20260927 21:31:52.764104 20006 tablet_service.cc:1514] Processing DeleteTablet for tablet 74f5fd23f6994c31948028c089aa17d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:31:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58498
I20260927 21:31:52.764724 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 19809
I20260927 21:31:52.773005 20249 tablet_replica.cc:330] T 74f5fd23f6994c31948028c089aa17d6 P a344fd139d98495eb26a51392651f9c0: stopping tablet replica
I20260927 21:31:52.774431 20249 raft_consensus.cc:2229] T 74f5fd23f6994c31948028c089aa17d6 P a344fd139d98495eb26a51392651f9c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:31:52.776268 20249 raft_consensus.cc:2258] T 74f5fd23f6994c31948028c089aa17d6 P a344fd139d98495eb26a51392651f9c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:31:52.778640 20248 tablet_replica.cc:330] T 74f5fd23f6994c31948028c089aa17d6 P b6879451b5714c539667c34a14edbb92: stopping tablet replica
I20260927 21:31:52.779109 20248 raft_consensus.cc:2229] T 74f5fd23f6994c31948028c089aa17d6 P b6879451b5714c539667c34a14edbb92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:31:52.779639 20248 raft_consensus.cc:2258] T 74f5fd23f6994c31948028c089aa17d6 P b6879451b5714c539667c34a14edbb92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:31:52.780426 20249 ts_tablet_manager.cc:1905] T 74f5fd23f6994c31948028c089aa17d6 P a344fd139d98495eb26a51392651f9c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:31:52.782161 20250 tablet_replica.cc:330] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346: stopping tablet replica
I20260927 21:31:52.782626 20250 raft_consensus.cc:2229] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:31:52.783337 20250 raft_consensus.cc:2258] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:31:52.785229 20250 ts_tablet_manager.cc:1905] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:31:52.790810 20249 ts_tablet_manager.cc:1918] T 74f5fd23f6994c31948028c089aa17d6 P a344fd139d98495eb26a51392651f9c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:31:52.790997 20249 log.cc:1196] T 74f5fd23f6994c31948028c089aa17d6 P a344fd139d98495eb26a51392651f9c0: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-0/wal/wals/74f5fd23f6994c31948028c089aa17d6
I20260927 21:31:52.791574 20249 ts_tablet_manager.cc:1939] T 74f5fd23f6994c31948028c089aa17d6 P a344fd139d98495eb26a51392651f9c0: Deleting consensus metadata
Sep 27 21:31:52 dist-test-slave-bxm0 krb5kdc[19703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 21:31:52.793031 20250 ts_tablet_manager.cc:1918] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:31:52.793292 20250 log.cc:1196] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-1/wal/wals/74f5fd23f6994c31948028c089aa17d6
I20260927 21:31:52.793406 20248 ts_tablet_manager.cc:1905] T 74f5fd23f6994c31948028c089aa17d6 P b6879451b5714c539667c34a14edbb92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:31:52.793970 20250 ts_tablet_manager.cc:1939] T 74f5fd23f6994c31948028c089aa17d6 P 6183a260ab2d4e05bef2b50019a07346: Deleting consensus metadata
I20260927 21:31:52.794570 19733 catalog_manager.cc:4841] TS a344fd139d98495eb26a51392651f9c0 (127.0.97.1:33159): tablet 74f5fd23f6994c31948028c089aa17d6 (table test-table [id=a7c909d61ef9417eb97925839954b87e]) successfully deleted
I20260927 21:31:52.795992 19733 catalog_manager.cc:4841] TS 6183a260ab2d4e05bef2b50019a07346 (127.0.97.2:39677): tablet 74f5fd23f6994c31948028c089aa17d6 (table test-table [id=a7c909d61ef9417eb97925839954b87e]) successfully deleted
I20260927 21:31:52.800390 20248 ts_tablet_manager.cc:1918] T 74f5fd23f6994c31948028c089aa17d6 P b6879451b5714c539667c34a14edbb92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:31:52.800555 20248 log.cc:1196] T 74f5fd23f6994c31948028c089aa17d6 P b6879451b5714c539667c34a14edbb92: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790544390748963-388-0/minicluster-data/ts-2/wal/wals/74f5fd23f6994c31948028c089aa17d6
I20260927 21:31:52.801038 20248 ts_tablet_manager.cc:1939] T 74f5fd23f6994c31948028c089aa17d6 P b6879451b5714c539667c34a14edbb92: Deleting consensus metadata
I20260927 21:31:52.801915 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:52.766070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37448 (local address 127.0.97.1:33159)
0927 21:31:52.766298 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:52.766342 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:52.779347 (+ 13005us) server_negotiation.cc:407] Connection header received
0927 21:31:52.779685 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:52.779693 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:52.779803 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:52.779885 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:52.783822 (+ 3937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.784973 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.786955 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.787606 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.787716 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:52.793280 (+ 5564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:52.793295 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:52.793325 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:52.793377 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:52.798028 (+ 4651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:52.798918 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:52.798928 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:52.798931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:52.799176 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:52.799661 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:52.799667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:52.799670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:52.800145 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:52.800462 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:52.801236 (+ 774us) server_negotiation.cc:299] Negotiation successful
0927 21:31:52.801475 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 21:31:52.802683 19733 catalog_manager.cc:4841] TS b6879451b5714c539667c34a14edbb92 (127.0.97.3:42927): tablet 74f5fd23f6994c31948028c089aa17d6 (table test-table [id=a7c909d61ef9417eb97925839954b87e]) successfully deleted
I20260927 21:31:52.803617 19932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:52.900446 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 19809
I20260927 21:31:52.918977 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 19943
Sep 27 21:31:52 dist-test-slave-bxm0 krb5kdc[19703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 21:31:52.936772 20223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:52.920176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58518 (local address 127.0.97.2:39677)
0927 21:31:52.920358 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:52.920363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:52.920377 (+ 14us) server_negotiation.cc:407] Connection header received
0927 21:31:52.920428 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:52.920432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:52.920488 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:52.920565 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:52.922286 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.923096 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.924916 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:52.925321 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:52.925415 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:52.930820 (+ 5405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:52.930835 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:52.930851 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:52.930894 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:52.934068 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:52.934870 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:52.934876 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:52.934878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:52.935012 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:52.935405 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:52.935408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:52.935410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:52.935742 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:52.936001 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:52.936300 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 21:31:52.936461 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:31:52.937654 20066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:53.053399 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 19943
I20260927 21:31:53.072516 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 20077
Sep 27 21:31:53 dist-test-slave-bxm0 krb5kdc[19703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 21:31:53.088569 20219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:53.073350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47944 (local address 127.0.97.3:42927)
0927 21:31:53.073547 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:53.073556 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:53.073577 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:31:53.073819 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:53.073829 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:53.073895 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:53.074005 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:53.075370 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:53.076112 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:53.077420 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:53.078026 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:53.078176 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:53.082875 (+ 4699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:53.082893 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:53.082909 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:53.082954 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:53.085662 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:53.086445 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:53.086457 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:53.086461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:53.086616 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:53.087006 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:53.087012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:53.087015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:53.087378 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:53.087669 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:53.088003 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 21:31:53.088227 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 21:31:53.089238 20200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:53.181242 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 20077
I20260927 21:31:53.197796 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 19716
I20260927 21:31:53.198873 19778 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:53.310194 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 19716
2026-09-27T21:31:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4366 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T21:31:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:31:53Z Disabled control of system clock
I20260927 21:31:53.386795 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41181
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:31:53.734855 20259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:53.735168 20259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:53.735250 20259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:53.742811 20259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:31:53.742934 20259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:53.742972 20259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:53.743021 20259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:31:53.743069 20259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:31:53.752985 20259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41181
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:53.754299 20259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:53.756039 20259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:53.767998 20264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:53.768054 20265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:53.767975 20267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:53.769004 20259 server_base.cc:1034] running on GCE node
I20260927 21:31:53.769872 20259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:53.771338 20259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:53.772607 20259 hybrid_clock.cc:648] HybridClock initialized: now 1790544713772555 us; error 43 us; skew 500 ppm
I20260927 21:31:53.772996 20259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:31:53.775516 20259 webserver.cc:460] Webserver started at http://127.0.97.62:39211/ using document root <none> and password file <none>
I20260927 21:31:53.776278 20259 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:53.776422 20259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:53.776757 20259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:31:53.779862 20259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "0d4289b3ed8847a99f90cc9a366d6aad"
format_stamp: "Formatted at 2026-09-27 21:31:53 on dist-test-slave-bxm0"
server_key: "0b820ce0ed8a9bd70fb19f5441303119"
server_key_iv: "d08e543d5b4f4e428d7bac7c85c17058"
server_key_version: "encryptionkey@0"
I20260927 21:31:53.780730 20259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "0d4289b3ed8847a99f90cc9a366d6aad"
format_stamp: "Formatted at 2026-09-27 21:31:53 on dist-test-slave-bxm0"
server_key: "0b820ce0ed8a9bd70fb19f5441303119"
server_key_iv: "d08e543d5b4f4e428d7bac7c85c17058"
server_key_version: "encryptionkey@0"
I20260927 21:31:53.786309 20259 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 21:31:53.789968 20273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:53.791383 20259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 21:31:53.791584 20259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "0d4289b3ed8847a99f90cc9a366d6aad"
format_stamp: "Formatted at 2026-09-27 21:31:53 on dist-test-slave-bxm0"
server_key: "0b820ce0ed8a9bd70fb19f5441303119"
server_key_iv: "d08e543d5b4f4e428d7bac7c85c17058"
server_key_version: "encryptionkey@0"
I20260927 21:31:53.791818 20259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:53.818984 20259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:53.819993 20259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:53.820327 20259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:53.839298 20259 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41181
I20260927 21:31:53.839318 20324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41181 every 8 connection(s)
I20260927 21:31:53.840706 20259 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:31:53.841881 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 20259
I20260927 21:31:53.842072 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:31:53.842507 388 external_mini_cluster.cc:1428] Setting key 21a826cac7a0b1fd259bb57e6b1a1b33
I20260927 21:31:53.845072 20325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:31:53.857674 20328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:53.844717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57134 (local address 127.0.97.62:41181)
0927 21:31:53.846390 (+ 1673us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:53.846400 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:53.846443 (+ 43us) server_negotiation.cc:407] Connection header received
0927 21:31:53.848056 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:53.848077 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:53.848545 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:53.848934 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:53.849540 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:53.850982 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:53.852949 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:53.853720 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:53.853856 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:53.855186 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:53.855195 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:53.855214 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:31:53.855802 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:53.855887 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:31:53.855909 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:53.856002 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:53.856243 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 21:31:53.856618 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1225,"thread_start_us":962,"threads_started":1}
I20260927 21:31:53.859097 20325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d4289b3ed8847a99f90cc9a366d6aad: Bootstrap starting.
I20260927 21:31:53.862519 20325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d4289b3ed8847a99f90cc9a366d6aad: Neither blocks nor log segments found. Creating new log.
I20260927 21:31:53.863592 20325 log.cc:824] T 00000000000000000000000000000000 P 0d4289b3ed8847a99f90cc9a366d6aad: Log is configured to *not* fsync() on all Append() calls
I20260927 21:31:53.866586 20325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d4289b3ed8847a99f90cc9a366d6aad: No bootstrap required, opened a new log
I20260927 21:31:53.874396 20325 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d4289b3ed8847a99f90cc9a366d6aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d4289b3ed8847a99f90cc9a366d6aad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41181 } }
I20260927 21:31:53.874796 20325 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d4289b3ed8847a99f90cc9a366d6aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:31:53.874924 20325 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d4289b3ed8847a99f90cc9a366d6aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d4289b3ed8847a99f90cc9a366d6aad, State: Initialized, Role: FOLLOWER
I20260927 21:31:53.875568 20325 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d4289b3ed8847a99f90cc9a366d6aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d4289b3ed8847a99f90cc9a366d6aad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41181 } }
I20260927 21:31:53.875828 20325 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d4289b3ed8847a99f90cc9a366d6aad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:31:53.875964 20325 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d4289b3ed8847a99f90cc9a366d6aad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:31:53.876178 20325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d4289b3ed8847a99f90cc9a366d6aad [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:31:53.878664 20325 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d4289b3ed8847a99f90cc9a366d6aad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d4289b3ed8847a99f90cc9a366d6aad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41181 } }
I20260927 21:31:53.879242 20325 leader_election.cc:302] T 00000000000000000000000000000000 P 0d4289b3ed8847a99f90cc9a366d6aad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d4289b3ed8847a99f90cc9a366d6aad; no voters:
I20260927 21:31:53.879873 20325 leader_election.cc:288] T 00000000000000000000000000000000 P 0d4289b3ed8847a99f90cc9a366d6aad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:31:53.880053 20330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d4289b3ed8847a99f90cc9a366d6aad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:31:53.880803 20330 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d4289b3ed8847a99f90cc9a366d6aad [term 1 LEADER]: Becoming Leader. State: Replica: 0d4289b3ed8847a99f90cc9a366d6aad, State: Running, Role: LEADER
I20260927 21:31:53.881373 20330 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d4289b3ed8847a99f90cc9a366d6aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d4289b3ed8847a99f90cc9a366d6aad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41181 } }
I20260927 21:31:53.882117 20325 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d4289b3ed8847a99f90cc9a366d6aad [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:31:53.884619 20332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d4289b3ed8847a99f90cc9a366d6aad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d4289b3ed8847a99f90cc9a366d6aad. Latest consensus state: current_term: 1 leader_uuid: "0d4289b3ed8847a99f90cc9a366d6aad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d4289b3ed8847a99f90cc9a366d6aad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41181 } } }
I20260927 21:31:53.885157 20332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d4289b3ed8847a99f90cc9a366d6aad [sys.catalog]: This master's current role is: LEADER
I20260927 21:31:53.885233 20331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d4289b3ed8847a99f90cc9a366d6aad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d4289b3ed8847a99f90cc9a366d6aad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d4289b3ed8847a99f90cc9a366d6aad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41181 } } }
I20260927 21:31:53.885540 20331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d4289b3ed8847a99f90cc9a366d6aad [sys.catalog]: This master's current role is: LEADER
I20260927 21:31:53.886410 20338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:31:53.894351 20338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:31:53.903964 20338 catalog_manager.cc:1347] Generated new cluster ID: 8bd2283f07c14048a8f68b3fa0a88eaf
I20260927 21:31:53.904100 20338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:31:53.930188 20338 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:31:53.931560 20338 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:31:53.938561 20338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d4289b3ed8847a99f90cc9a366d6aad: Generated new TSK 0
I20260927 21:31:53.939536 20338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:31:53.957962 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41181
--builtin_ntp_servers=127.0.97.20:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:31:54.308943 20349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:54.309248 20349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:54.309337 20349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:54.316825 20349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:54.316946 20349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:54.317066 20349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 21:31:54.326843 20349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:54.328158 20349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:54.329923 20349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:54.343623 20357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:54.343683 20355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:54.343683 20354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:54.344347 20349 server_base.cc:1034] running on GCE node
I20260927 21:31:54.344967 20349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:54.346019 20349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:54.347267 20349 hybrid_clock.cc:648] HybridClock initialized: now 1790544714347217 us; error 45 us; skew 500 ppm
I20260927 21:31:54.347620 20349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:31:54.350107 20349 webserver.cc:460] Webserver started at http://127.0.97.1:42605/ using document root <none> and password file <none>
I20260927 21:31:54.350907 20349 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:54.351043 20349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:54.351388 20349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:31:54.354583 20349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-0/data/instance:
uuid: "eb12ffd0f2ed4050b55a344c53adfc98"
format_stamp: "Formatted at 2026-09-27 21:31:54 on dist-test-slave-bxm0"
server_key: "33ff2e465b598a21894005a397239034"
server_key_iv: "e2e470e15004349c9a0132cf7a7f967e"
server_key_version: "encryptionkey@0"
I20260927 21:31:54.355432 20349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-0/wal/instance:
uuid: "eb12ffd0f2ed4050b55a344c53adfc98"
format_stamp: "Formatted at 2026-09-27 21:31:54 on dist-test-slave-bxm0"
server_key: "33ff2e465b598a21894005a397239034"
server_key_iv: "e2e470e15004349c9a0132cf7a7f967e"
server_key_version: "encryptionkey@0"
I20260927 21:31:54.361703 20349 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.002s
I20260927 21:31:54.365724 20363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:54.367120 20349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:31:54.367317 20349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-0/wal
uuid: "eb12ffd0f2ed4050b55a344c53adfc98"
format_stamp: "Formatted at 2026-09-27 21:31:54 on dist-test-slave-bxm0"
server_key: "33ff2e465b598a21894005a397239034"
server_key_iv: "e2e470e15004349c9a0132cf7a7f967e"
server_key_version: "encryptionkey@0"
I20260927 21:31:54.367552 20349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:54.382750 20349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:54.383834 20349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:54.384188 20349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:54.385186 20349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:31:54.387012 20349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:31:54.387130 20349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:54.387274 20349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:31:54.387359 20349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:54.421427 20349 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32993
I20260927 21:31:54.421482 20475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32993 every 8 connection(s)
I20260927 21:31:54.422798 20349 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-0/data/info.pb
I20260927 21:31:54.424502 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 20349
I20260927 21:31:54.424676 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-0/wal/instance
I20260927 21:31:54.425271 388 external_mini_cluster.cc:1428] Setting key 19d5046c7173a00ba36a2f89bd09ba1e
I20260927 21:31:54.428279 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41181
--builtin_ntp_servers=127.0.97.20:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:31:54.436699 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:54.425710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40113 (local address 127.0.97.62:41181)
0927 21:31:54.426213 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:54.426217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:54.427619 (+ 1402us) server_negotiation.cc:407] Connection header received
0927 21:31:54.429534 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:54.429539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:54.429659 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:54.429787 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:54.431300 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:54.432042 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:54.433668 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:54.434081 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:54.434220 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:54.435704 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:54.435713 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:54.435719 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:31:54.435749 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:54.435802 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:31:54.435820 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:54.435927 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:54.436128 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 21:31:54.436247 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":269,"threads_started":1}
I20260927 21:31:54.437561 20479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:54.426361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41181 (local address 127.0.97.1:40113)
0927 21:31:54.427465 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:54.427511 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:54.429267 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:54.430025 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:54.430034 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:54.430187 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:31:54.431135 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:54.431153 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:54.432196 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:54.432201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:54.433492 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:54.433512 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:54.434312 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:54.434322 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:54.434594 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:54.435368 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:31:54.435370 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:31:54.435412 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:31:54.435492 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:31:54.435925 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:31:54.435933 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:31:54.435939 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 21:31:54.436080 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 21:31:54.436397 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":297,"threads_started":1}
I20260927 21:31:54.439324 20476 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41181
I20260927 21:31:54.439647 20476 heartbeater.cc:464] Registering TS with master...
I20260927 21:31:54.440447 20476 heartbeater.cc:511] Master 127.0.97.62:41181 requested a full tablet report, sending...
I20260927 21:31:54.442515 20290 ts_manager.cc:194] Registered new tserver with Master: eb12ffd0f2ed4050b55a344c53adfc98 (127.0.97.1:32993)
I20260927 21:31:54.443977 20290 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:40113
W20260927 21:31:54.759785 20480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:54.760087 20480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:54.760223 20480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:54.767661 20480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:54.767777 20480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:54.767952 20480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 21:31:54.777661 20480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:54.779008 20480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:54.780898 20480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:54.794907 20485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:54.794755 20486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:54.794737 20488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:54.796029 20480 server_base.cc:1034] running on GCE node
I20260927 21:31:54.796766 20480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:54.797883 20480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:54.799103 20480 hybrid_clock.cc:648] HybridClock initialized: now 1790544714799044 us; error 51 us; skew 500 ppm
I20260927 21:31:54.799443 20480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:31:54.802289 20480 webserver.cc:460] Webserver started at http://127.0.97.2:35489/ using document root <none> and password file <none>
I20260927 21:31:54.803191 20480 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:54.803313 20480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:54.803663 20480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:31:54.806665 20480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-1/data/instance:
uuid: "6c55b236426e48ab8bf7dbe1655ec261"
format_stamp: "Formatted at 2026-09-27 21:31:54 on dist-test-slave-bxm0"
server_key: "c6da529874d6b6f6e4ee3ab7d5137df6"
server_key_iv: "0c8f14681679dafb84bae72d5c16e330"
server_key_version: "encryptionkey@0"
I20260927 21:31:54.807495 20480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6c55b236426e48ab8bf7dbe1655ec261"
format_stamp: "Formatted at 2026-09-27 21:31:54 on dist-test-slave-bxm0"
server_key: "c6da529874d6b6f6e4ee3ab7d5137df6"
server_key_iv: "0c8f14681679dafb84bae72d5c16e330"
server_key_version: "encryptionkey@0"
I20260927 21:31:54.812666 20480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 21:31:54.816636 20494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:54.818142 20480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:31:54.818317 20480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-1/wal
uuid: "6c55b236426e48ab8bf7dbe1655ec261"
format_stamp: "Formatted at 2026-09-27 21:31:54 on dist-test-slave-bxm0"
server_key: "c6da529874d6b6f6e4ee3ab7d5137df6"
server_key_iv: "0c8f14681679dafb84bae72d5c16e330"
server_key_version: "encryptionkey@0"
I20260927 21:31:54.818542 20480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:54.835821 20480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:54.836731 20480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:54.837029 20480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:54.838059 20480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:31:54.839741 20480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:31:54.839871 20480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:54.840009 20480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:31:54.840097 20480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:54.874925 20480 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44037
I20260927 21:31:54.875021 20606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44037 every 8 connection(s)
I20260927 21:31:54.876319 20480 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-1/data/info.pb
I20260927 21:31:54.884819 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 20480
I20260927 21:31:54.885007 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-1/wal/instance
I20260927 21:31:54.885429 388 external_mini_cluster.cc:1428] Setting key ecf078b25efc9cdccec4109dff3957dc
I20260927 21:31:54.888556 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41181
--builtin_ntp_servers=127.0.97.20:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:31:54.890237 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:54.879004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45075 (local address 127.0.97.62:41181)
0927 21:31:54.879223 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:54.879230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:54.880793 (+ 1563us) server_negotiation.cc:407] Connection header received
0927 21:31:54.882687 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:54.882691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:54.882765 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:54.882937 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:54.884020 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:54.884956 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:54.887099 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:54.887667 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:54.887791 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:54.889233 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:54.889242 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:54.889251 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:31:54.889279 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:54.889324 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:31:54.889342 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:54.889440 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:54.889723 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 21:31:54.889837 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 21:31:54.891188 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:54.879601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41181 (local address 127.0.97.2:45075)
0927 21:31:54.880619 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:54.880661 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:54.882423 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:54.883190 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:54.883207 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:54.883298 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:31:54.883871 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:54.883888 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:54.885143 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:54.885148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:54.886964 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:54.886978 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:54.887877 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:54.887882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:54.888099 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:54.888909 (+ 810us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:31:54.888911 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:31:54.888954 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:31:54.889038 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:31:54.889442 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:31:54.889450 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:31:54.889455 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:31:54.889701 (+ 246us) client_negotiation.cc:240] Negotiation successful
0927 21:31:54.890024 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":303,"threads_started":1}
I20260927 21:31:54.892658 20607 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41181
I20260927 21:31:54.892988 20607 heartbeater.cc:464] Registering TS with master...
I20260927 21:31:54.893989 20607 heartbeater.cc:511] Master 127.0.97.62:41181 requested a full tablet report, sending...
I20260927 21:31:54.895679 20290 ts_manager.cc:194] Registered new tserver with Master: 6c55b236426e48ab8bf7dbe1655ec261 (127.0.97.2:44037)
I20260927 21:31:54.896605 20290 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:45075
W20260927 21:31:55.227133 20610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:55.227432 20610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:55.227519 20610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:55.234949 20610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:55.235064 20610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:55.235250 20610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 21:31:55.244493 20610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:55.245895 20610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:55.247633 20610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:55.260430 20615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:55.260535 20618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:55.260540 20616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:55.261675 20610 server_base.cc:1034] running on GCE node
I20260927 21:31:55.262565 20610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:55.263609 20610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:55.264858 20610 hybrid_clock.cc:648] HybridClock initialized: now 1790544715264814 us; error 47 us; skew 500 ppm
I20260927 21:31:55.265219 20610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:31:55.267982 20610 webserver.cc:460] Webserver started at http://127.0.97.3:39565/ using document root <none> and password file <none>
I20260927 21:31:55.268746 20610 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:55.268877 20610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:55.269243 20610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:31:55.272598 20610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-2/data/instance:
uuid: "c6beb617672b45a2ad3a5b1a7d042cff"
format_stamp: "Formatted at 2026-09-27 21:31:55 on dist-test-slave-bxm0"
server_key: "ebfe3feb88a03b296d69f8ed699bfc8d"
server_key_iv: "a8d0efacb23c006d0c2adfe42520d01a"
server_key_version: "encryptionkey@0"
I20260927 21:31:55.273514 20610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c6beb617672b45a2ad3a5b1a7d042cff"
format_stamp: "Formatted at 2026-09-27 21:31:55 on dist-test-slave-bxm0"
server_key: "ebfe3feb88a03b296d69f8ed699bfc8d"
server_key_iv: "a8d0efacb23c006d0c2adfe42520d01a"
server_key_version: "encryptionkey@0"
I20260927 21:31:55.279209 20610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 21:31:55.283106 20624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:55.285037 20610 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 21:31:55.285295 20610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-2/wal
uuid: "c6beb617672b45a2ad3a5b1a7d042cff"
format_stamp: "Formatted at 2026-09-27 21:31:55 on dist-test-slave-bxm0"
server_key: "ebfe3feb88a03b296d69f8ed699bfc8d"
server_key_iv: "a8d0efacb23c006d0c2adfe42520d01a"
server_key_version: "encryptionkey@0"
I20260927 21:31:55.285578 20610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:55.318193 20610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:55.319072 20610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:55.319351 20610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:55.320345 20610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:31:55.322098 20610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:31:55.322216 20610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:55.322350 20610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:31:55.322432 20610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:55.356427 20610 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35087
I20260927 21:31:55.356516 20736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35087 every 8 connection(s)
I20260927 21:31:55.357724 20610 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-2/data/info.pb
I20260927 21:31:55.365841 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 20610
I20260927 21:31:55.366032 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-2/wal/instance
I20260927 21:31:55.366559 388 external_mini_cluster.cc:1428] Setting key c1d415c1a28a11034743d2c743b1d6a7
I20260927 21:31:55.370466 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:55.360274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54285 (local address 127.0.97.62:41181)
0927 21:31:55.360466 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:55.360471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:55.361974 (+ 1503us) server_negotiation.cc:407] Connection header received
0927 21:31:55.363719 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:55.363724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:55.363793 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:55.363887 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:55.364861 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.365696 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.367676 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.368097 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.368225 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:55.369584 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:55.369592 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:55.369601 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:31:55.369663 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:55.369704 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:31:55.369717 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:55.369812 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:55.370030 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 21:31:55.370147 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 21:31:55.371557 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:55.360805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41181 (local address 127.0.97.3:54285)
0927 21:31:55.361820 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:55.361857 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:55.363452 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:55.364083 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:55.364091 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:55.364173 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:31:55.364722 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:55.364737 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.365854 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.365862 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:55.367507 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:55.367518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.368323 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.368330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:55.368567 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:55.369271 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:31:55.369274 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:31:55.369313 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:31:55.369388 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:31:55.369812 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:31:55.369825 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:31:55.369836 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 21:31:55.370021 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 21:31:55.370397 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":291,"threads_started":1}
I20260927 21:31:55.372982 20737 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41181
I20260927 21:31:55.373245 20737 heartbeater.cc:464] Registering TS with master...
I20260927 21:31:55.374014 20737 heartbeater.cc:511] Master 127.0.97.62:41181 requested a full tablet report, sending...
I20260927 21:31:55.375341 20290 ts_manager.cc:194] Registered new tserver with Master: c6beb617672b45a2ad3a5b1a7d042cff (127.0.97.3:35087)
I20260927 21:31:55.376106 20290 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:54285
I20260927 21:31:55.381441 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:31:55.391517 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:55.385376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57150 (local address 127.0.97.62:41181)
0927 21:31:55.385674 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:55.385684 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:55.386152 (+ 468us) server_negotiation.cc:407] Connection header received
0927 21:31:55.386299 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:55.386303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:55.386378 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:55.386460 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:55.386887 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.387660 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.389205 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.389737 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.389835 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:55.390950 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:55.390955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:55.390958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:31:55.390977 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:55.391008 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:31:55.391020 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:55.391093 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:55.391220 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 21:31:55.391299 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 21:31:55.395427 20290 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:31:55.402487 20290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:31:55.438612 20751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:55.415685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32993 (local address 127.0.0.1:40310)
0927 21:31:55.428299 (+ 12614us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:55.428320 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:55.428477 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:55.430102 (+ 1625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:55.430110 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:55.430141 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:55.430512 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:55.430529 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.432507 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.432512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:55.435146 (+ 2634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:55.435161 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.437485 (+ 2324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.437492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:55.437751 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:55.437826 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 21:31:55.437986 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 21:31:55.438233 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12477,"thread_start_us":13739,"threads_started":1}
I20260927 21:31:55.439616 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:55.416323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40310 (local address 127.0.97.1:32993)
0927 21:31:55.426341 (+ 10018us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:55.426351 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:55.428691 (+ 2340us) server_negotiation.cc:407] Connection header received
0927 21:31:55.428791 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:55.428799 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:55.429061 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:55.429232 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:55.430726 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.432316 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.435377 (+ 3061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.436941 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.437073 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:55.437357 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:55.439061 (+ 1704us) server_negotiation.cc:299] Negotiation successful
0927 21:31:55.439251 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9784,"thread_start_us":9655,"threads_started":1}
I20260927 21:31:55.441840 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:55.415281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35087 (local address 127.0.0.1:46992)
0927 21:31:55.428208 (+ 12927us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:55.428260 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:55.428472 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:55.433191 (+ 4719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:55.433201 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:55.433235 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:55.433674 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:55.433687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.435585 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.435590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:55.438808 (+ 3218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:55.438822 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.441032 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.441039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:55.441228 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:55.441252 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 21:31:55.441348 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 21:31:55.441483 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12547,"thread_start_us":15080,"threads_started":1}
I20260927 21:31:55.441987 20750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:55.415611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46992 (local address 127.0.97.3:35087)
0927 21:31:55.432244 (+ 16633us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:55.432254 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:55.432299 (+ 45us) server_negotiation.cc:407] Connection header received
0927 21:31:55.432400 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:55.432408 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:55.432699 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:55.432913 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:55.433897 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.435418 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.439036 (+ 3618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.440566 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.440694 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:55.440964 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:55.441419 (+ 455us) server_negotiation.cc:299] Negotiation successful
0927 21:31:55.441585 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16450,"thread_start_us":16301,"threads_started":1}
I20260927 21:31:55.444456 20411 tablet_service.cc:1467] Processing CreateTablet for tablet 59cf530819c845edb751dfa079656675 (DEFAULT_TABLE table=test-table [id=d7a566733ed844e6b148588154cd863b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:31:55.445199 20672 tablet_service.cc:1467] Processing CreateTablet for tablet 59cf530819c845edb751dfa079656675 (DEFAULT_TABLE table=test-table [id=d7a566733ed844e6b148588154cd863b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:31:55.445907 20411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59cf530819c845edb751dfa079656675. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:31:55.446970 20476 heartbeater.cc:503] Master 127.0.97.62:41181 was elected leader, sending a full tablet report...
I20260927 21:31:55.447304 20672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59cf530819c845edb751dfa079656675. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:31:55.449136 20753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:55.420721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44037 (local address 127.0.0.1:53880)
0927 21:31:55.440858 (+ 20137us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:55.440881 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:55.441014 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:55.441693 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:55.441697 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:55.441724 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:55.442085 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:55.442101 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.444203 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.444208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:55.446177 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:55.446193 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.448353 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.448361 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:55.448549 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:55.448583 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 21:31:55.448668 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 21:31:55.448809 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19992,"thread_start_us":19836,"threads_started":1}
I20260927 21:31:55.450022 20754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:55.425116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53880 (local address 127.0.97.2:44037)
0927 21:31:55.440120 (+ 15004us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:55.440133 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:55.441054 (+ 921us) server_negotiation.cc:407] Connection header received
0927 21:31:55.441168 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:55.441177 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:55.441461 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:55.442317 (+ 856us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:55.442572 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.444036 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.446423 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.447854 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.447989 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:55.448214 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:55.449474 (+ 1260us) server_negotiation.cc:299] Negotiation successful
0927 21:31:55.449683 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14720,"thread_start_us":15005,"threads_started":1}
I20260927 21:31:55.453827 20542 tablet_service.cc:1467] Processing CreateTablet for tablet 59cf530819c845edb751dfa079656675 (DEFAULT_TABLE table=test-table [id=d7a566733ed844e6b148588154cd863b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:31:55.455356 20542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59cf530819c845edb751dfa079656675. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:31:55.458971 20755 tablet_bootstrap.cc:492] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98: Bootstrap starting.
I20260927 21:31:55.461418 20756 tablet_bootstrap.cc:492] T 59cf530819c845edb751dfa079656675 P c6beb617672b45a2ad3a5b1a7d042cff: Bootstrap starting.
I20260927 21:31:55.462881 20755 tablet_bootstrap.cc:654] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98: Neither blocks nor log segments found. Creating new log.
I20260927 21:31:55.464202 20755 log.cc:824] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98: Log is configured to *not* fsync() on all Append() calls
I20260927 21:31:55.465152 20756 tablet_bootstrap.cc:654] T 59cf530819c845edb751dfa079656675 P c6beb617672b45a2ad3a5b1a7d042cff: Neither blocks nor log segments found. Creating new log.
I20260927 21:31:55.466408 20756 log.cc:824] T 59cf530819c845edb751dfa079656675 P c6beb617672b45a2ad3a5b1a7d042cff: Log is configured to *not* fsync() on all Append() calls
I20260927 21:31:55.468006 20755 tablet_bootstrap.cc:492] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98: No bootstrap required, opened a new log
I20260927 21:31:55.468339 20755 ts_tablet_manager.cc:1397] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260927 21:31:55.469579 20759 tablet_bootstrap.cc:492] T 59cf530819c845edb751dfa079656675 P 6c55b236426e48ab8bf7dbe1655ec261: Bootstrap starting.
I20260927 21:31:55.469916 20756 tablet_bootstrap.cc:492] T 59cf530819c845edb751dfa079656675 P c6beb617672b45a2ad3a5b1a7d042cff: No bootstrap required, opened a new log
I20260927 21:31:55.470268 20756 ts_tablet_manager.cc:1397] T 59cf530819c845edb751dfa079656675 P c6beb617672b45a2ad3a5b1a7d042cff: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 21:31:55.472841 20759 tablet_bootstrap.cc:654] T 59cf530819c845edb751dfa079656675 P 6c55b236426e48ab8bf7dbe1655ec261: Neither blocks nor log segments found. Creating new log.
I20260927 21:31:55.473863 20759 log.cc:824] T 59cf530819c845edb751dfa079656675 P 6c55b236426e48ab8bf7dbe1655ec261: Log is configured to *not* fsync() on all Append() calls
I20260927 21:31:55.476441 20759 tablet_bootstrap.cc:492] T 59cf530819c845edb751dfa079656675 P 6c55b236426e48ab8bf7dbe1655ec261: No bootstrap required, opened a new log
I20260927 21:31:55.476722 20759 ts_tablet_manager.cc:1397] T 59cf530819c845edb751dfa079656675 P 6c55b236426e48ab8bf7dbe1655ec261: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 21:31:55.480120 20755 raft_consensus.cc:348] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6beb617672b45a2ad3a5b1a7d042cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35087 } } peers { permanent_uuid: "eb12ffd0f2ed4050b55a344c53adfc98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32993 } } peers { permanent_uuid: "6c55b236426e48ab8bf7dbe1655ec261" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44037 } }
I20260927 21:31:55.480675 20755 raft_consensus.cc:374] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:31:55.480832 20755 raft_consensus.cc:729] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb12ffd0f2ed4050b55a344c53adfc98, State: Initialized, Role: FOLLOWER
I20260927 21:31:55.480803 20756 raft_consensus.cc:348] T 59cf530819c845edb751dfa079656675 P c6beb617672b45a2ad3a5b1a7d042cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6beb617672b45a2ad3a5b1a7d042cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35087 } } peers { permanent_uuid: "eb12ffd0f2ed4050b55a344c53adfc98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32993 } } peers { permanent_uuid: "6c55b236426e48ab8bf7dbe1655ec261" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44037 } }
I20260927 21:31:55.481356 20756 raft_consensus.cc:374] T 59cf530819c845edb751dfa079656675 P c6beb617672b45a2ad3a5b1a7d042cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:31:55.481498 20756 raft_consensus.cc:729] T 59cf530819c845edb751dfa079656675 P c6beb617672b45a2ad3a5b1a7d042cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6beb617672b45a2ad3a5b1a7d042cff, State: Initialized, Role: FOLLOWER
I20260927 21:31:55.481526 20755 consensus_queue.cc:260] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6beb617672b45a2ad3a5b1a7d042cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35087 } } peers { permanent_uuid: "eb12ffd0f2ed4050b55a344c53adfc98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32993 } } peers { permanent_uuid: "6c55b236426e48ab8bf7dbe1655ec261" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44037 } }
I20260927 21:31:55.482124 20756 consensus_queue.cc:260] T 59cf530819c845edb751dfa079656675 P c6beb617672b45a2ad3a5b1a7d042cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6beb617672b45a2ad3a5b1a7d042cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35087 } } peers { permanent_uuid: "eb12ffd0f2ed4050b55a344c53adfc98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32993 } } peers { permanent_uuid: "6c55b236426e48ab8bf7dbe1655ec261" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44037 } }
I20260927 21:31:55.483912 20755 ts_tablet_manager.cc:1428] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260927 21:31:55.485731 20756 ts_tablet_manager.cc:1428] T 59cf530819c845edb751dfa079656675 P c6beb617672b45a2ad3a5b1a7d042cff: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260927 21:31:55.486241 20737 heartbeater.cc:503] Master 127.0.97.62:41181 was elected leader, sending a full tablet report...
I20260927 21:31:55.486079 20759 raft_consensus.cc:348] T 59cf530819c845edb751dfa079656675 P 6c55b236426e48ab8bf7dbe1655ec261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6beb617672b45a2ad3a5b1a7d042cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35087 } } peers { permanent_uuid: "eb12ffd0f2ed4050b55a344c53adfc98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32993 } } peers { permanent_uuid: "6c55b236426e48ab8bf7dbe1655ec261" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44037 } }
I20260927 21:31:55.486560 20759 raft_consensus.cc:374] T 59cf530819c845edb751dfa079656675 P 6c55b236426e48ab8bf7dbe1655ec261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:31:55.486673 20759 raft_consensus.cc:729] T 59cf530819c845edb751dfa079656675 P 6c55b236426e48ab8bf7dbe1655ec261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c55b236426e48ab8bf7dbe1655ec261, State: Initialized, Role: FOLLOWER
I20260927 21:31:55.487295 20759 consensus_queue.cc:260] T 59cf530819c845edb751dfa079656675 P 6c55b236426e48ab8bf7dbe1655ec261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6beb617672b45a2ad3a5b1a7d042cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35087 } } peers { permanent_uuid: "eb12ffd0f2ed4050b55a344c53adfc98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32993 } } peers { permanent_uuid: "6c55b236426e48ab8bf7dbe1655ec261" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44037 } }
I20260927 21:31:55.488641 20607 heartbeater.cc:503] Master 127.0.97.62:41181 was elected leader, sending a full tablet report...
I20260927 21:31:55.490447 20759 ts_tablet_manager.cc:1428] T 59cf530819c845edb751dfa079656675 P 6c55b236426e48ab8bf7dbe1655ec261: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
W20260927 21:31:55.609401 20738 tablet.cc:2369] T 59cf530819c845edb751dfa079656675 P c6beb617672b45a2ad3a5b1a7d042cff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:31:55.628506 20608 tablet.cc:2369] T 59cf530819c845edb751dfa079656675 P 6c55b236426e48ab8bf7dbe1655ec261: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:31:55.628988 20761 raft_consensus.cc:482] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:31:55.629293 20761 raft_consensus.cc:504] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6beb617672b45a2ad3a5b1a7d042cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35087 } } peers { permanent_uuid: "eb12ffd0f2ed4050b55a344c53adfc98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32993 } } peers { permanent_uuid: "6c55b236426e48ab8bf7dbe1655ec261" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44037 } }
I20260927 21:31:55.631105 20761 leader_election.cc:288] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6beb617672b45a2ad3a5b1a7d042cff (127.0.97.3:35087), 6c55b236426e48ab8bf7dbe1655ec261 (127.0.97.2:44037)
I20260927 21:31:55.638043 20754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:55.631823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57561 (local address 127.0.97.2:44037)
0927 21:31:55.631993 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:55.632000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:55.632346 (+ 346us) server_negotiation.cc:407] Connection header received
0927 21:31:55.632579 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:55.632587 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:55.632680 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:55.632770 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:55.633593 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.634335 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.636361 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.637121 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.637258 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:55.637361 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:55.637659 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 21:31:55.637786 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:31:55.638110 20750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:55.631476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37997 (local address 127.0.97.3:35087)
0927 21:31:55.631690 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:55.631698 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:55.632663 (+ 965us) server_negotiation.cc:407] Connection header received
0927 21:31:55.632785 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:55.632788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:55.632850 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:55.632942 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:31:55.633573 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.634343 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.636376 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.637177 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.637324 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:55.637426 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:55.637745 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 21:31:55.637873 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 21:31:55.638161 20764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:55.631415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44037 (local address 127.0.97.1:57561)
0927 21:31:55.632196 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:55.632234 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:55.632417 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:55.633065 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:55.633068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:55.633102 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:55.633437 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:55.633445 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.634483 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.634489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:55.636161 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:55.636174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.637319 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.637325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:55.637484 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:55.637508 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 21:31:55.637611 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 21:31:55.637758 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":100,"thread_start_us":481,"threads_started":1}
I20260927 21:31:55.638255 20765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:55.631555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35087 (local address 127.0.97.1:37997)
0927 21:31:55.632539 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:55.632555 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:31:55.632677 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:31:55.632986 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:31:55.632989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:31:55.633014 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:31:55.633377 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:55.633384 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.634488 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.634493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:55.636172 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:31:55.636183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.637375 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.637381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:31:55.637550 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:55.637572 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 21:31:55.637716 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 21:31:55.637841 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"mutex_wait_us":74,"thread_start_us":656,"threads_started":1}
I20260927 21:31:55.639226 20562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59cf530819c845edb751dfa079656675" candidate_uuid: "eb12ffd0f2ed4050b55a344c53adfc98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c55b236426e48ab8bf7dbe1655ec261" is_pre_election: true
I20260927 21:31:55.639489 20692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59cf530819c845edb751dfa079656675" candidate_uuid: "eb12ffd0f2ed4050b55a344c53adfc98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6beb617672b45a2ad3a5b1a7d042cff" is_pre_election: true
I20260927 21:31:55.639784 20562 raft_consensus.cc:2454] T 59cf530819c845edb751dfa079656675 P 6c55b236426e48ab8bf7dbe1655ec261 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb12ffd0f2ed4050b55a344c53adfc98 in term 0.
I20260927 21:31:55.639988 20692 raft_consensus.cc:2454] T 59cf530819c845edb751dfa079656675 P c6beb617672b45a2ad3a5b1a7d042cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb12ffd0f2ed4050b55a344c53adfc98 in term 0.
I20260927 21:31:55.640511 20364 leader_election.cc:302] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c55b236426e48ab8bf7dbe1655ec261, eb12ffd0f2ed4050b55a344c53adfc98; no voters:
I20260927 21:31:55.640856 20761 raft_consensus.cc:2781] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:31:55.640975 20761 raft_consensus.cc:482] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:31:55.641089 20761 raft_consensus.cc:3037] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:31:55.643348 20761 raft_consensus.cc:504] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6beb617672b45a2ad3a5b1a7d042cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35087 } } peers { permanent_uuid: "eb12ffd0f2ed4050b55a344c53adfc98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32993 } } peers { permanent_uuid: "6c55b236426e48ab8bf7dbe1655ec261" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44037 } }
I20260927 21:31:55.644038 20761 leader_election.cc:288] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98 [CANDIDATE]: Term 1 election: Requested vote from peers c6beb617672b45a2ad3a5b1a7d042cff (127.0.97.3:35087), 6c55b236426e48ab8bf7dbe1655ec261 (127.0.97.2:44037)
I20260927 21:31:55.644500 20692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59cf530819c845edb751dfa079656675" candidate_uuid: "eb12ffd0f2ed4050b55a344c53adfc98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6beb617672b45a2ad3a5b1a7d042cff"
I20260927 21:31:55.644631 20562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59cf530819c845edb751dfa079656675" candidate_uuid: "eb12ffd0f2ed4050b55a344c53adfc98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c55b236426e48ab8bf7dbe1655ec261"
I20260927 21:31:55.644726 20692 raft_consensus.cc:3037] T 59cf530819c845edb751dfa079656675 P c6beb617672b45a2ad3a5b1a7d042cff [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:31:55.644842 20562 raft_consensus.cc:3037] T 59cf530819c845edb751dfa079656675 P 6c55b236426e48ab8bf7dbe1655ec261 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:31:55.648000 20562 raft_consensus.cc:2454] T 59cf530819c845edb751dfa079656675 P 6c55b236426e48ab8bf7dbe1655ec261 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb12ffd0f2ed4050b55a344c53adfc98 in term 1.
I20260927 21:31:55.648000 20692 raft_consensus.cc:2454] T 59cf530819c845edb751dfa079656675 P c6beb617672b45a2ad3a5b1a7d042cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb12ffd0f2ed4050b55a344c53adfc98 in term 1.
I20260927 21:31:55.648456 20364 leader_election.cc:302] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6c55b236426e48ab8bf7dbe1655ec261, c6beb617672b45a2ad3a5b1a7d042cff, eb12ffd0f2ed4050b55a344c53adfc98; no voters:
I20260927 21:31:55.648808 20761 raft_consensus.cc:2781] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:31:55.649253 20761 raft_consensus.cc:686] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98 [term 1 LEADER]: Becoming Leader. State: Replica: eb12ffd0f2ed4050b55a344c53adfc98, State: Running, Role: LEADER
I20260927 21:31:55.649834 20761 consensus_queue.cc:237] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6beb617672b45a2ad3a5b1a7d042cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35087 } } peers { permanent_uuid: "eb12ffd0f2ed4050b55a344c53adfc98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32993 } } peers { permanent_uuid: "6c55b236426e48ab8bf7dbe1655ec261" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44037 } }
I20260927 21:31:55.654611 20289 catalog_manager.cc:5495] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb12ffd0f2ed4050b55a344c53adfc98 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "eb12ffd0f2ed4050b55a344c53adfc98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6beb617672b45a2ad3a5b1a7d042cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb12ffd0f2ed4050b55a344c53adfc98" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c55b236426e48ab8bf7dbe1655ec261" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44037 } health_report { overall_health: UNKNOWN } } }
W20260927 21:31:55.675230 20477 tablet.cc:2369] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:31:55.723024 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:55.717839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40314 (local address 127.0.97.1:32993)
0927 21:31:55.718050 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:55.718054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:55.718071 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:31:55.718242 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:55.718245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:55.718358 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:55.718478 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:31:55.718868 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.719539 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.720874 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.721276 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.721364 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:55.721892 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:31:55.722403 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:31:55.722520 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:55.722657 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 21:31:55.722720 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 21:31:55.731101 20692 raft_consensus.cc:1261] T 59cf530819c845edb751dfa079656675 P c6beb617672b45a2ad3a5b1a7d042cff [term 1 FOLLOWER]: Refusing update from remote peer eb12ffd0f2ed4050b55a344c53adfc98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:31:55.731094 20562 raft_consensus.cc:1261] T 59cf530819c845edb751dfa079656675 P 6c55b236426e48ab8bf7dbe1655ec261 [term 1 FOLLOWER]: Refusing update from remote peer eb12ffd0f2ed4050b55a344c53adfc98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:31:55.732054 20761 consensus_queue.cc:1035] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c55b236426e48ab8bf7dbe1655ec261" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:31:55.732347 20766 consensus_queue.cc:1035] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6beb617672b45a2ad3a5b1a7d042cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:31:55.759516 20290 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:31:55.759841 20290 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:31:55.763592 20290 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d4289b3ed8847a99f90cc9a366d6aad: Sending DeleteTablet for 3 replicas of tablet 59cf530819c845edb751dfa079656675
I20260927 21:31:55.764760 20411 tablet_service.cc:1514] Processing DeleteTablet for tablet 59cf530819c845edb751dfa079656675 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:31:55 UTC) from {username='slave'} at 127.0.0.1:40310
I20260927 21:31:55.764780 20672 tablet_service.cc:1514] Processing DeleteTablet for tablet 59cf530819c845edb751dfa079656675 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:31:55 UTC) from {username='slave'} at 127.0.0.1:46992
I20260927 21:31:55.765069 20542 tablet_service.cc:1514] Processing DeleteTablet for tablet 59cf530819c845edb751dfa079656675 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:31:55 UTC) from {username='slave'} at 127.0.0.1:53880
I20260927 21:31:55.767251 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 20349
I20260927 21:31:55.777863 20781 tablet_replica.cc:330] T 59cf530819c845edb751dfa079656675 P 6c55b236426e48ab8bf7dbe1655ec261: stopping tablet replica
I20260927 21:31:55.778319 20779 tablet_replica.cc:330] T 59cf530819c845edb751dfa079656675 P c6beb617672b45a2ad3a5b1a7d042cff: stopping tablet replica
I20260927 21:31:55.778476 20781 raft_consensus.cc:2229] T 59cf530819c845edb751dfa079656675 P 6c55b236426e48ab8bf7dbe1655ec261 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:31:55.778959 20779 raft_consensus.cc:2229] T 59cf530819c845edb751dfa079656675 P c6beb617672b45a2ad3a5b1a7d042cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:31:55.779112 20781 raft_consensus.cc:2258] T 59cf530819c845edb751dfa079656675 P 6c55b236426e48ab8bf7dbe1655ec261 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:31:55.779944 20780 tablet_replica.cc:330] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98: stopping tablet replica
I20260927 21:31:55.780392 20780 raft_consensus.cc:2229] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:31:55.781129 20780 raft_consensus.cc:2258] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:31:55.781559 20781 ts_tablet_manager.cc:1905] T 59cf530819c845edb751dfa079656675 P 6c55b236426e48ab8bf7dbe1655ec261: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:31:55.783519 20780 ts_tablet_manager.cc:1905] T 59cf530819c845edb751dfa079656675 P eb12ffd0f2ed4050b55a344c53adfc98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:31:55.785177 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:55.768881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40322 (local address 127.0.97.1:32993)
0927 21:31:55.769113 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:55.769122 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:55.774921 (+ 5799us) server_negotiation.cc:407] Connection header received
0927 21:31:55.776005 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:55.776015 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:55.776149 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:55.776240 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:55.777747 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.779053 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.781030 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.781758 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.781881 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:55.783406 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:55.783419 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:55.783436 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:31:55.784198 (+ 762us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:55.784255 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:31:55.784281 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:55.784365 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:55.784655 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 21:31:55.784774 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 21:31:55.786154 20779 raft_consensus.cc:2258] T 59cf530819c845edb751dfa079656675 P c6beb617672b45a2ad3a5b1a7d042cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:31:55.786341 20471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:55.788436 20779 ts_tablet_manager.cc:1905] T 59cf530819c845edb751dfa079656675 P c6beb617672b45a2ad3a5b1a7d042cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:31:55.790371 20781 ts_tablet_manager.cc:1918] T 59cf530819c845edb751dfa079656675 P 6c55b236426e48ab8bf7dbe1655ec261: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:31:55.790558 20781 log.cc:1196] T 59cf530819c845edb751dfa079656675 P 6c55b236426e48ab8bf7dbe1655ec261: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-1/wal/wals/59cf530819c845edb751dfa079656675
I20260927 21:31:55.791179 20781 ts_tablet_manager.cc:1939] T 59cf530819c845edb751dfa079656675 P 6c55b236426e48ab8bf7dbe1655ec261: Deleting consensus metadata
I20260927 21:31:55.793294 20274 catalog_manager.cc:4841] TS 6c55b236426e48ab8bf7dbe1655ec261 (127.0.97.2:44037): tablet 59cf530819c845edb751dfa079656675 (table test-table [id=d7a566733ed844e6b148588154cd863b]) successfully deleted
I20260927 21:31:55.795696 20779 ts_tablet_manager.cc:1918] T 59cf530819c845edb751dfa079656675 P c6beb617672b45a2ad3a5b1a7d042cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:31:55.795894 20779 log.cc:1196] T 59cf530819c845edb751dfa079656675 P c6beb617672b45a2ad3a5b1a7d042cff: Deleting WAL directory at /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790544390748963-388-0/minicluster-data/ts-2/wal/wals/59cf530819c845edb751dfa079656675
I20260927 21:31:55.796347 20779 ts_tablet_manager.cc:1939] T 59cf530819c845edb751dfa079656675 P c6beb617672b45a2ad3a5b1a7d042cff: Deleting consensus metadata
I20260927 21:31:55.797821 20275 catalog_manager.cc:4841] TS c6beb617672b45a2ad3a5b1a7d042cff (127.0.97.3:35087): tablet 59cf530819c845edb751dfa079656675 (table test-table [id=d7a566733ed844e6b148588154cd863b]) successfully deleted
I20260927 21:31:55.888684 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 20349
I20260927 21:31:55.907768 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 20480
W20260927 21:31:55.908017 20276 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:32993 (error 108)
I20260927 21:31:55.909873 20753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:55.909332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32993 (local address 127.0.0.1:40330)
0927 21:31:55.909531 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:55.909685 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 21:31:55.910118 20276 catalog_manager.cc:4584] TS eb12ffd0f2ed4050b55a344c53adfc98 (127.0.97.1:32993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 59cf530819c845edb751dfa079656675: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32993: connect: Connection refused (error 111)
I20260927 21:31:55.914942 20754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:55.908612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53886 (local address 127.0.97.2:44037)
0927 21:31:55.908798 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:55.908803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:55.908921 (+ 118us) server_negotiation.cc:407] Connection header received
0927 21:31:55.909170 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:55.909179 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:55.909334 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:55.909447 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:55.909953 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.910792 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.912000 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:55.912405 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:55.912500 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:55.913518 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:55.913526 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:55.913541 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:31:55.914214 (+ 673us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:55.914261 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:31:55.914281 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:55.914380 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:55.914492 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 21:31:55.914582 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 21:31:55.915668 20602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:55.963547 20753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:55.963096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32993 (local address 127.0.0.1:40342)
0927 21:31:55.963314 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:55.963403 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 21:31:55.964586 20753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:55.964255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32993 (local address 127.0.0.1:40348)
0927 21:31:55.964401 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:55.964474 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 21:31:56.005869 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 20480
I20260927 21:31:56.023587 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 20610
I20260927 21:31:56.025321 20753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:56.024826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32993 (local address 127.0.0.1:40350)
0927 21:31:56.025041 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:56.025134 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 21:31:56.026412 20753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:56.026035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32993 (local address 127.0.0.1:40356)
0927 21:31:56.026195 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:56.026279 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 21:31:56.030932 20750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:56.024326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47004 (local address 127.0.97.3:35087)
0927 21:31:56.024509 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:56.024514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:56.024587 (+ 73us) server_negotiation.cc:407] Connection header received
0927 21:31:56.024773 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:56.024776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:56.024903 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:56.025029 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:56.025417 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:56.026508 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:56.027767 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:56.028168 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:56.028312 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:56.029429 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:56.029439 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:56.029455 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:31:56.030157 (+ 702us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:56.030206 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:31:56.030232 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:56.030357 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:56.030499 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 21:31:56.030606 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 21:31:56.031562 20732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:56.107162 20753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:56.106564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32993 (local address 127.0.0.1:40364)
0927 21:31:56.106832 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:56.106942 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260927 21:31:56.108350 20753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:56.107936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32993 (local address 127.0.0.1:40368)
0927 21:31:56.108153 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 21:31:56.108234 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260927 21:31:56.120679 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 20610
I20260927 21:31:56.137970 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 20259
I20260927 21:31:56.139102 20320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:56.251444 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 20259
2026-09-27T21:31:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2940 ms)
[----------] 6 tests from EncryptionPolicyTest (22224 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:31:56 dist-test-slave-bxm0 krb5kdc[20790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:31:56 dist-test-slave-bxm0 krb5kdc[20790](info): set up 2 sockets
Sep 27 21:31:56 dist-test-slave-bxm0 krb5kdc[20790](info): commencing operation
krb5kdc: starting...
W20260927 21:31:57.362221 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:31:57 dist-test-slave-bxm0 krb5kdc[20790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:31:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:31:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:31:57.625896 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45723
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:31:57.962437 20803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:31:57.962735 20803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:31:57.962929 20803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:31:57.970458 20803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:31:57.970582 20803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:31:57.970688 20803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:31:57.970806 20803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:31:57.970890 20803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:31:57.981086 20803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45723
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:31:57.982437 20803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:31:57.983700 20803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:31:57.995962 20808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:57.996001 20811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:31:57.995923 20809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:31:57.996887 20803 server_base.cc:1034] running on GCE node
I20260927 21:31:57.997903 20803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:31:57.999326 20803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:31:58.000586 20803 hybrid_clock.cc:648] HybridClock initialized: now 1790544718000486 us; error 55 us; skew 500 ppm
I20260927 21:31:58.000983 20803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:31:58 dist-test-slave-bxm0 krb5kdc[20790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:31:58.016970 20803 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:31:58.018918 20803 webserver.cc:460] Webserver started at http://127.0.0.1:38457/ using document root <none> and password file <none>
I20260927 21:31:58.019706 20803 fs_manager.cc:359] Metadata directory not provided
I20260927 21:31:58.019909 20803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:31:58.020254 20803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:31:58.023547 20803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "e7513fca73264b3c8dba0f5ebf1dd214"
format_stamp: "Formatted at 2026-09-27 21:31:58 on dist-test-slave-bxm0"
server_key: "c78d3f71de21e439d3eddfac38b0ff11"
server_key_iv: "264ee9ae6fb4418c2e2ab7595f67b88d"
server_key_version: "encryptionkey@0"
I20260927 21:31:58.024376 20803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "e7513fca73264b3c8dba0f5ebf1dd214"
format_stamp: "Formatted at 2026-09-27 21:31:58 on dist-test-slave-bxm0"
server_key: "c78d3f71de21e439d3eddfac38b0ff11"
server_key_iv: "264ee9ae6fb4418c2e2ab7595f67b88d"
server_key_version: "encryptionkey@0"
I20260927 21:31:58.029980 20803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260927 21:31:58.033713 20818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:31:58.035161 20803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:31:58.035364 20803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "e7513fca73264b3c8dba0f5ebf1dd214"
format_stamp: "Formatted at 2026-09-27 21:31:58 on dist-test-slave-bxm0"
server_key: "c78d3f71de21e439d3eddfac38b0ff11"
server_key_iv: "264ee9ae6fb4418c2e2ab7595f67b88d"
server_key_version: "encryptionkey@0"
I20260927 21:31:58.035570 20803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:31:58.050683 20803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:31:58.054868 20803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:31:58.055166 20803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:31:58.074314 20803 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45723
I20260927 21:31:58.074354 20869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45723 every 8 connection(s)
I20260927 21:31:58.075728 20803 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:31:58.079833 20870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:31:58.083518 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 20803
I20260927 21:31:58.083750 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:31:58.084407 388 external_mini_cluster.cc:1428] Setting key eda7155bf40bce13f9c7f586129ad53b
I20260927 21:31:58.093902 20870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7513fca73264b3c8dba0f5ebf1dd214: Bootstrap starting.
I20260927 21:31:58.097986 20870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7513fca73264b3c8dba0f5ebf1dd214: Neither blocks nor log segments found. Creating new log.
I20260927 21:31:58.099058 20870 log.cc:824] T 00000000000000000000000000000000 P e7513fca73264b3c8dba0f5ebf1dd214: Log is configured to *not* fsync() on all Append() calls
Sep 27 21:31:58 dist-test-slave-bxm0 krb5kdc[20790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 21:31:58.102105 20870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7513fca73264b3c8dba0f5ebf1dd214: No bootstrap required, opened a new log
I20260927 21:31:58.109164 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:58.086543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37313 (local address 127.0.0.1:45723)
0927 21:31:58.087362 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:58.087381 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:58.087435 (+ 54us) server_negotiation.cc:407] Connection header received
0927 21:31:58.089399 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:58.089433 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:58.090061 (+ 628us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:58.090541 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:58.091918 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:58.093995 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:58.095490 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:58.096627 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:58.096815 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:58.101284 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:58.101293 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:58.101309 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:58.101349 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:58.104628 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:58.105510 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:58.105517 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:58.105527 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:58.105761 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:58.106250 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:58.106256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:58.106263 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:58.106839 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:58.107193 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:58.107712 (+ 519us) server_negotiation.cc:299] Negotiation successful
0927 21:31:58.108057 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":343,"threads_started":1}
I20260927 21:31:58.110307 20870 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7513fca73264b3c8dba0f5ebf1dd214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7513fca73264b3c8dba0f5ebf1dd214" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45723 } }
I20260927 21:31:58.110790 20870 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7513fca73264b3c8dba0f5ebf1dd214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:31:58.110914 20870 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7513fca73264b3c8dba0f5ebf1dd214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7513fca73264b3c8dba0f5ebf1dd214, State: Initialized, Role: FOLLOWER
I20260927 21:31:58.111593 20870 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7513fca73264b3c8dba0f5ebf1dd214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7513fca73264b3c8dba0f5ebf1dd214" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45723 } }
I20260927 21:31:58.111912 20870 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7513fca73264b3c8dba0f5ebf1dd214 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:31:58.112062 20870 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7513fca73264b3c8dba0f5ebf1dd214 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:31:58.112262 20870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7513fca73264b3c8dba0f5ebf1dd214 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:31:58.113758 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:31:58.114946 20870 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7513fca73264b3c8dba0f5ebf1dd214 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7513fca73264b3c8dba0f5ebf1dd214" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45723 } }
I20260927 21:31:58.115515 20870 leader_election.cc:302] T 00000000000000000000000000000000 P e7513fca73264b3c8dba0f5ebf1dd214 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7513fca73264b3c8dba0f5ebf1dd214; no voters:
I20260927 21:31:58.116242 20870 leader_election.cc:288] T 00000000000000000000000000000000 P e7513fca73264b3c8dba0f5ebf1dd214 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:31:58.116359 20875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7513fca73264b3c8dba0f5ebf1dd214 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:31:58.119302 20870 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7513fca73264b3c8dba0f5ebf1dd214 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:31:58.123448 20875 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7513fca73264b3c8dba0f5ebf1dd214 [term 1 LEADER]: Becoming Leader. State: Replica: e7513fca73264b3c8dba0f5ebf1dd214, State: Running, Role: LEADER
I20260927 21:31:58.124289 20875 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7513fca73264b3c8dba0f5ebf1dd214 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7513fca73264b3c8dba0f5ebf1dd214" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45723 } }
I20260927 21:31:58.127977 20878 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7513fca73264b3c8dba0f5ebf1dd214 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7513fca73264b3c8dba0f5ebf1dd214. Latest consensus state: current_term: 1 leader_uuid: "e7513fca73264b3c8dba0f5ebf1dd214" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7513fca73264b3c8dba0f5ebf1dd214" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45723 } } }
I20260927 21:31:58.128527 20878 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7513fca73264b3c8dba0f5ebf1dd214 [sys.catalog]: This master's current role is: LEADER
I20260927 21:31:58.129212 20879 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7513fca73264b3c8dba0f5ebf1dd214 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7513fca73264b3c8dba0f5ebf1dd214" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7513fca73264b3c8dba0f5ebf1dd214" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45723 } } }
I20260927 21:31:58.129969 20879 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7513fca73264b3c8dba0f5ebf1dd214 [sys.catalog]: This master's current role is: LEADER
I20260927 21:31:58.134697 20887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:31:58.140357 20887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:31:58.156950 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:31:58.131466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54359 (local address 127.0.0.1:45723)
0927 21:31:58.131698 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 21:31:58.131707 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:31:58.134645 (+ 2938us) server_negotiation.cc:407] Connection header received
0927 21:31:58.134964 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:31:58.134969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:31:58.135051 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:31:58.135164 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:31:58.137518 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:58.139196 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:58.142007 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:31:58.143003 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:31:58.143154 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:31:58.147584 (+ 4430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:31:58.147592 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:31:58.147600 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:31:58.147661 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:31:58.152584 (+ 4923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:58.154041 (+ 1457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:58.154050 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:58.154056 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:58.154261 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:31:58.154960 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:31:58.154970 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:31:58.154975 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:31:58.155414 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:31:58.155669 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 21:31:58.156372 (+ 703us) server_negotiation.cc:299] Negotiation successful
0927 21:31:58.156571 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 21:31:58.158712 20887 catalog_manager.cc:1347] Generated new cluster ID: dabde5a192664b128295166d16ab0a5c
I20260927 21:31:58.159361 20887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:31:58.179817 20887 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:31:58.181185 20887 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:31:58.190579 20887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7513fca73264b3c8dba0f5ebf1dd214: Generated new TSK 0
I20260927 21:31:58.191429 20887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:31:58.206790 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 20803
I20260927 21:31:58.208614 20865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:31:58.305747 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 20803
W20260927 21:31:58.323429 20788 connection.cc:535] client connection to 127.0.0.1:45723 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45723): Connection reset by peer (error 104)
2026-09-27T21:31:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2048 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:31:58 dist-test-slave-bxm0 krb5kdc[20903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:31:58 dist-test-slave-bxm0 krb5kdc[20903](info): set up 2 sockets
Sep 27 21:31:58 dist-test-slave-bxm0 krb5kdc[20903](info): commencing operation
krb5kdc: starting...
W20260927 21:31:59.418010 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:31:59 dist-test-slave-bxm0 krb5kdc[20903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:31:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:31:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:31:59.684723 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35467
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:32:00.031172 20916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:00.031653 20916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:00.031852 20916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:00.044137 20916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:32:00.044268 20916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:00.044317 20916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:00.044371 20916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:32:00.044414 20916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:32:00.054669 20916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35467
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:00.055934 20916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:00.057173 20916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:00.069320 20924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:00.069386 20922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:00.069585 20921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:00.070582 20916 server_base.cc:1034] running on GCE node
I20260927 21:32:00.071482 20916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:00.072954 20916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:00.074184 20916 hybrid_clock.cc:648] HybridClock initialized: now 1790544720074137 us; error 59 us; skew 500 ppm
I20260927 21:32:00.074544 20916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:00 dist-test-slave-bxm0 krb5kdc[20903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:00.077795 20916 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:32:00.079275 20916 webserver.cc:460] Webserver started at http://127.0.0.1:40719/ using document root <none> and password file <none>
I20260927 21:32:00.079945 20916 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:00.080046 20916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:00.080310 20916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:00.083365 20916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "e13a493ea83b42c4b419f114d4bcce32"
format_stamp: "Formatted at 2026-09-27 21:32:00 on dist-test-slave-bxm0"
server_key: "17671a7fb84b74cdca0fe5ad217e1fb0"
server_key_iv: "d0cfcd40a486a9e73c200bffbe667fdd"
server_key_version: "encryptionkey@0"
I20260927 21:32:00.084081 20916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "e13a493ea83b42c4b419f114d4bcce32"
format_stamp: "Formatted at 2026-09-27 21:32:00 on dist-test-slave-bxm0"
server_key: "17671a7fb84b74cdca0fe5ad217e1fb0"
server_key_iv: "d0cfcd40a486a9e73c200bffbe667fdd"
server_key_version: "encryptionkey@0"
I20260927 21:32:00.089378 20916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:32:00.092872 20931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:00.094363 20916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:32:00.094498 20916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "e13a493ea83b42c4b419f114d4bcce32"
format_stamp: "Formatted at 2026-09-27 21:32:00 on dist-test-slave-bxm0"
server_key: "17671a7fb84b74cdca0fe5ad217e1fb0"
server_key_iv: "d0cfcd40a486a9e73c200bffbe667fdd"
server_key_version: "encryptionkey@0"
I20260927 21:32:00.094666 20916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:00.108743 20916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:00.112859 20916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:00.113104 20916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:00.132272 20916 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35467
I20260927 21:32:00.132296 20982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35467 every 8 connection(s)
I20260927 21:32:00.133745 20916 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:32:00.137426 20983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:32:00.139755 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 20916
I20260927 21:32:00.139968 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:32:00.140508 388 external_mini_cluster.cc:1428] Setting key 3d4d305592615ee7e025cf870b54359a
I20260927 21:32:00.149940 20983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e13a493ea83b42c4b419f114d4bcce32: Bootstrap starting.
I20260927 21:32:00.153661 20983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e13a493ea83b42c4b419f114d4bcce32: Neither blocks nor log segments found. Creating new log.
I20260927 21:32:00.154251 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:00.142395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54455 (local address 127.0.0.1:35467)
0927 21:32:00.143323 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:00.143337 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:00.143379 (+ 42us) server_negotiation.cc:407] Connection header received
0927 21:32:00.145055 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:00.145084 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:00.145474 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:00.145852 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:00.146442 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:00.147775 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:00.149698 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:00.150468 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:00.150605 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:00.151869 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:00.151877 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:00.151890 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:32:00.152427 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:00.152512 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:32:00.152535 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:00.152647 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:00.152948 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 21:32:00.153200 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":398,"threads_started":1}
I20260927 21:32:00.154867 20983 log.cc:824] T 00000000000000000000000000000000 P e13a493ea83b42c4b419f114d4bcce32: Log is configured to *not* fsync() on all Append() calls
I20260927 21:32:00.158036 20983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e13a493ea83b42c4b419f114d4bcce32: No bootstrap required, opened a new log
I20260927 21:32:00.158351 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:32:00.167059 20983 raft_consensus.cc:348] T 00000000000000000000000000000000 P e13a493ea83b42c4b419f114d4bcce32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13a493ea83b42c4b419f114d4bcce32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35467 } }
I20260927 21:32:00.167652 20983 raft_consensus.cc:374] T 00000000000000000000000000000000 P e13a493ea83b42c4b419f114d4bcce32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:32:00.167783 20983 raft_consensus.cc:729] T 00000000000000000000000000000000 P e13a493ea83b42c4b419f114d4bcce32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e13a493ea83b42c4b419f114d4bcce32, State: Initialized, Role: FOLLOWER
I20260927 21:32:00.168505 20983 consensus_queue.cc:260] T 00000000000000000000000000000000 P e13a493ea83b42c4b419f114d4bcce32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13a493ea83b42c4b419f114d4bcce32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35467 } }
I20260927 21:32:00.168802 20983 raft_consensus.cc:388] T 00000000000000000000000000000000 P e13a493ea83b42c4b419f114d4bcce32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:32:00.168947 20983 raft_consensus.cc:482] T 00000000000000000000000000000000 P e13a493ea83b42c4b419f114d4bcce32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:32:00.169149 20983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e13a493ea83b42c4b419f114d4bcce32 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:32:00.170018 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:00.162701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41875 (local address 127.0.0.1:35467)
0927 21:32:00.162899 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:00.162907 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:00.163263 (+ 356us) server_negotiation.cc:407] Connection header received
0927 21:32:00.163477 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:00.163481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:00.163541 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:00.163651 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:00.164154 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:00.164921 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:00.166956 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:00.167593 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:00.167756 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:00.169240 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:00.169264 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:00.169270 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:32:00.169296 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:00.169347 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:32:00.169370 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:00.169455 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:00.169722 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 21:32:00.169769 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:32:00.172425 20983 raft_consensus.cc:504] T 00000000000000000000000000000000 P e13a493ea83b42c4b419f114d4bcce32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13a493ea83b42c4b419f114d4bcce32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35467 } }
I20260927 21:32:00.173038 20983 leader_election.cc:302] T 00000000000000000000000000000000 P e13a493ea83b42c4b419f114d4bcce32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e13a493ea83b42c4b419f114d4bcce32; no voters:
I20260927 21:32:00.173672 20983 leader_election.cc:288] T 00000000000000000000000000000000 P e13a493ea83b42c4b419f114d4bcce32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:32:00.173898 20993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e13a493ea83b42c4b419f114d4bcce32 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:32:00.174914 20993 raft_consensus.cc:686] T 00000000000000000000000000000000 P e13a493ea83b42c4b419f114d4bcce32 [term 1 LEADER]: Becoming Leader. State: Replica: e13a493ea83b42c4b419f114d4bcce32, State: Running, Role: LEADER
I20260927 21:32:00.175575 20993 consensus_queue.cc:237] T 00000000000000000000000000000000 P e13a493ea83b42c4b419f114d4bcce32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13a493ea83b42c4b419f114d4bcce32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35467 } }
I20260927 21:32:00.176681 20983 sys_catalog.cc:564] T 00000000000000000000000000000000 P e13a493ea83b42c4b419f114d4bcce32 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:32:00.178387 20994 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13a493ea83b42c4b419f114d4bcce32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e13a493ea83b42c4b419f114d4bcce32. Latest consensus state: current_term: 1 leader_uuid: "e13a493ea83b42c4b419f114d4bcce32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13a493ea83b42c4b419f114d4bcce32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35467 } } }
I20260927 21:32:00.178542 20995 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13a493ea83b42c4b419f114d4bcce32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e13a493ea83b42c4b419f114d4bcce32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13a493ea83b42c4b419f114d4bcce32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35467 } } }
I20260927 21:32:00.178881 20995 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13a493ea83b42c4b419f114d4bcce32 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:00.178902 20994 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13a493ea83b42c4b419f114d4bcce32 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:00.179725 20998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:32:00.184412 20998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:32:00.194276 20998 catalog_manager.cc:1347] Generated new cluster ID: 6e8e4d954f1c4cf1bd9187cc8269edfe
I20260927 21:32:00.194391 20998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:32:00.207361 20998 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:32:00.208698 20998 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:32:00.221143 20998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e13a493ea83b42c4b419f114d4bcce32: Generated new TSK 0
I20260927 21:32:00.222290 20998 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:32:00.240553 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 20916
W20260927 21:32:00.241926 20978 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54455
W20260927 21:32:00.242621 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 20916: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:32:00.242766 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 20916
W20260927 21:32:00.259979 20901 connection.cc:535] client connection to 127.0.0.1:35467 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35467): Connection reset by peer (error 104)
2026-09-27T21:32:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1936 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:32:00 dist-test-slave-bxm0 krb5kdc[21014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:32:00 dist-test-slave-bxm0 krb5kdc[21014](info): set up 2 sockets
Sep 27 21:32:00 dist-test-slave-bxm0 krb5kdc[21014](info): commencing operation
krb5kdc: starting...
W20260927 21:32:01.345996 388 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:32:01 dist-test-slave-bxm0 krb5kdc[21014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:32:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:32:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:32:01.601379 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45149
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:32:01.939589 21027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:01.939986 21027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:01.940173 21027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:01.947573 21027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:32:01.947763 21027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:01.947885 21027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:01.948009 21027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:32:01.948108 21027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:32:01.958446 21027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45149
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:01.959897 21027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:01.961339 21027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:01.974320 21032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:01.974450 21035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:01.974237 21033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:01.975450 21027 server_base.cc:1034] running on GCE node
I20260927 21:32:01.976253 21027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:01.977697 21027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:01.978937 21027 hybrid_clock.cc:648] HybridClock initialized: now 1790544721978877 us; error 48 us; skew 500 ppm
I20260927 21:32:01.979317 21027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:01 dist-test-slave-bxm0 krb5kdc[21014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:01.982525 21027 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:32:01.984119 21027 webserver.cc:460] Webserver started at http://127.0.0.1:37577/ using document root <none> and password file <none>
I20260927 21:32:01.984807 21027 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:01.984944 21027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:01.985265 21027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:01.988279 21027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "cfbd8e69b5ad4f58a13413d9fc884285"
format_stamp: "Formatted at 2026-09-27 21:32:01 on dist-test-slave-bxm0"
server_key: "c0a3124826109e1b87620b062ee1d53c"
server_key_iv: "fae6d9bf9d7f274dfecd5e8f02c398b2"
server_key_version: "encryptionkey@0"
I20260927 21:32:01.989055 21027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "cfbd8e69b5ad4f58a13413d9fc884285"
format_stamp: "Formatted at 2026-09-27 21:32:01 on dist-test-slave-bxm0"
server_key: "c0a3124826109e1b87620b062ee1d53c"
server_key_iv: "fae6d9bf9d7f274dfecd5e8f02c398b2"
server_key_version: "encryptionkey@0"
I20260927 21:32:01.994210 21027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260927 21:32:01.997699 21042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:01.999017 21027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:32:01.999207 21027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "cfbd8e69b5ad4f58a13413d9fc884285"
format_stamp: "Formatted at 2026-09-27 21:32:01 on dist-test-slave-bxm0"
server_key: "c0a3124826109e1b87620b062ee1d53c"
server_key_iv: "fae6d9bf9d7f274dfecd5e8f02c398b2"
server_key_version: "encryptionkey@0"
I20260927 21:32:01.999426 21027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:02.005324 21027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:02.009418 21027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:02.009738 21027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:02.028261 21027 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45149
I20260927 21:32:02.028302 21093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45149 every 8 connection(s)
I20260927 21:32:02.029759 21027 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:32:02.033699 21094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:32:02.035224 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 21027
I20260927 21:32:02.035419 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:32:02.035983 388 external_mini_cluster.cc:1428] Setting key ea8938620c3ab431ad48212c04cbff16
I20260927 21:32:02.043532 21097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:02.037809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37741 (local address 127.0.0.1:45149)
0927 21:32:02.038635 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:02.038646 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:02.038692 (+ 46us) server_negotiation.cc:407] Connection header received
0927 21:32:02.040334 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:02.040355 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:02.040691 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:02.041004 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:02.042038 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:02.042044 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:02.042056 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:32:02.042069 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:02.042127 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:32:02.042145 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:02.042353 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:02.042451 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 21:32:02.042653 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":390,"threads_started":1}
I20260927 21:32:02.047190 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:32:02.047252 21094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfbd8e69b5ad4f58a13413d9fc884285: Bootstrap starting.
I20260927 21:32:02.051103 21094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfbd8e69b5ad4f58a13413d9fc884285: Neither blocks nor log segments found. Creating new log.
I20260927 21:32:02.052423 21094 log.cc:824] T 00000000000000000000000000000000 P cfbd8e69b5ad4f58a13413d9fc884285: Log is configured to *not* fsync() on all Append() calls
I20260927 21:32:02.055096 21097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:02.052543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52321 (local address 127.0.0.1:45149)
0927 21:32:02.052729 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:02.052735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:02.053119 (+ 384us) server_negotiation.cc:407] Connection header received
0927 21:32:02.053320 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:02.053324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:02.053383 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:02.053497 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:02.054457 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:02.054462 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:02.054467 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:32:02.054488 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:02.054534 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:32:02.054549 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:02.054664 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:02.054768 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 21:32:02.054830 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 21:32:02.055380 21094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfbd8e69b5ad4f58a13413d9fc884285: No bootstrap required, opened a new log
I20260927 21:32:02.063603 21094 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfbd8e69b5ad4f58a13413d9fc884285 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfbd8e69b5ad4f58a13413d9fc884285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45149 } }
I20260927 21:32:02.064019 21094 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfbd8e69b5ad4f58a13413d9fc884285 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:32:02.064133 21094 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfbd8e69b5ad4f58a13413d9fc884285 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfbd8e69b5ad4f58a13413d9fc884285, State: Initialized, Role: FOLLOWER
I20260927 21:32:02.064711 21094 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfbd8e69b5ad4f58a13413d9fc884285 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfbd8e69b5ad4f58a13413d9fc884285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45149 } }
I20260927 21:32:02.064967 21094 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfbd8e69b5ad4f58a13413d9fc884285 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:32:02.065120 21094 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfbd8e69b5ad4f58a13413d9fc884285 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:32:02.065294 21094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfbd8e69b5ad4f58a13413d9fc884285 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:32:02.067561 21094 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfbd8e69b5ad4f58a13413d9fc884285 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfbd8e69b5ad4f58a13413d9fc884285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45149 } }
I20260927 21:32:02.068069 21094 leader_election.cc:302] T 00000000000000000000000000000000 P cfbd8e69b5ad4f58a13413d9fc884285 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfbd8e69b5ad4f58a13413d9fc884285; no voters:
I20260927 21:32:02.068763 21094 leader_election.cc:288] T 00000000000000000000000000000000 P cfbd8e69b5ad4f58a13413d9fc884285 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:32:02.068914 21104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfbd8e69b5ad4f58a13413d9fc884285 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:32:02.069600 21104 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfbd8e69b5ad4f58a13413d9fc884285 [term 1 LEADER]: Becoming Leader. State: Replica: cfbd8e69b5ad4f58a13413d9fc884285, State: Running, Role: LEADER
I20260927 21:32:02.070111 21104 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfbd8e69b5ad4f58a13413d9fc884285 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfbd8e69b5ad4f58a13413d9fc884285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45149 } }
I20260927 21:32:02.071158 21094 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfbd8e69b5ad4f58a13413d9fc884285 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:32:02.073282 21106 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfbd8e69b5ad4f58a13413d9fc884285 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfbd8e69b5ad4f58a13413d9fc884285. Latest consensus state: current_term: 1 leader_uuid: "cfbd8e69b5ad4f58a13413d9fc884285" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfbd8e69b5ad4f58a13413d9fc884285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45149 } } }
I20260927 21:32:02.073240 21105 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfbd8e69b5ad4f58a13413d9fc884285 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfbd8e69b5ad4f58a13413d9fc884285" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfbd8e69b5ad4f58a13413d9fc884285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45149 } } }
I20260927 21:32:02.073776 21106 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfbd8e69b5ad4f58a13413d9fc884285 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:02.073791 21105 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfbd8e69b5ad4f58a13413d9fc884285 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:02.075670 21112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:32:02.081211 21112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:32:02.089601 21112 catalog_manager.cc:1347] Generated new cluster ID: 2a5e0ff7de0f401fb9f3b9dc2a984d85
I20260927 21:32:02.089756 21112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:32:02.115899 21112 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:32:02.116344 21112 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:32:02.124305 21112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfbd8e69b5ad4f58a13413d9fc884285: Generated new TSK 0
I20260927 21:32:02.125113 21112 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:32:02.132865 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 21027
W20260927 21:32:02.133991 21089 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37741
W20260927 21:32:02.134684 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 21027: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:32:02.134850 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 21027
2026-09-27T21:32:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1891 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:32:02 dist-test-slave-bxm0 krb5kdc[21125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:32:02 dist-test-slave-bxm0 krb5kdc[21125](info): set up 2 sockets
Sep 27 21:32:02 dist-test-slave-bxm0 krb5kdc[21125](info): commencing operation
krb5kdc: starting...
W20260927 21:32:03.253958 388 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:32:03 dist-test-slave-bxm0 krb5kdc[21125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:32:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:32:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:32:03.512673 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46481
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:32:03.858687 21138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:03.859158 21138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:03.859337 21138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:03.866883 21138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:32:03.867071 21138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:03.867161 21138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:03.867254 21138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:32:03.867339 21138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:32:03.877369 21138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:03.878836 21138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:03.880226 21138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:03.892920 21144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:03.892896 21146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:03.893143 21143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:03.899152 21138 server_base.cc:1034] running on GCE node
I20260927 21:32:03.900067 21138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:03.901528 21138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:03.902727 21138 hybrid_clock.cc:648] HybridClock initialized: now 1790544723902668 us; error 55 us; skew 500 ppm
I20260927 21:32:03.903081 21138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:03 dist-test-slave-bxm0 krb5kdc[21125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:03.906199 21138 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:32:03.907590 21138 webserver.cc:460] Webserver started at http://127.0.0.1:46101/ using document root <none> and password file <none>
I20260927 21:32:03.908234 21138 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:03.908329 21138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:03.908586 21138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:03.911687 21138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "eb6a341ea8e64267af6d36d1ce9b9616"
format_stamp: "Formatted at 2026-09-27 21:32:03 on dist-test-slave-bxm0"
server_key: "d4fd4231a258bb4b7767ea2479f6bcde"
server_key_iv: "59f427143193eb1cb13c766d7caf0a68"
server_key_version: "encryptionkey@0"
I20260927 21:32:03.912490 21138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "eb6a341ea8e64267af6d36d1ce9b9616"
format_stamp: "Formatted at 2026-09-27 21:32:03 on dist-test-slave-bxm0"
server_key: "d4fd4231a258bb4b7767ea2479f6bcde"
server_key_iv: "59f427143193eb1cb13c766d7caf0a68"
server_key_version: "encryptionkey@0"
I20260927 21:32:03.918059 21138 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260927 21:32:03.921473 21153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:03.922804 21138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:32:03.922989 21138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "eb6a341ea8e64267af6d36d1ce9b9616"
format_stamp: "Formatted at 2026-09-27 21:32:03 on dist-test-slave-bxm0"
server_key: "d4fd4231a258bb4b7767ea2479f6bcde"
server_key_iv: "59f427143193eb1cb13c766d7caf0a68"
server_key_version: "encryptionkey@0"
I20260927 21:32:03.923189 21138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:03.943148 21138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:03.947289 21138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:03.947593 21138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:03.966395 21138 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46481
I20260927 21:32:03.966430 21204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46481 every 8 connection(s)
I20260927 21:32:03.967957 21138 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:32:03.971669 21205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:32:03.977102 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 21138
I20260927 21:32:03.977303 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:32:03.977813 388 external_mini_cluster.cc:1428] Setting key fed7681b887291615d4dc00e53dc96f4
I20260927 21:32:03.982796 21205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb6a341ea8e64267af6d36d1ce9b9616: Bootstrap starting.
I20260927 21:32:03.986582 21205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb6a341ea8e64267af6d36d1ce9b9616: Neither blocks nor log segments found. Creating new log.
I20260927 21:32:03.987828 21205 log.cc:824] T 00000000000000000000000000000000 P eb6a341ea8e64267af6d36d1ce9b9616: Log is configured to *not* fsync() on all Append() calls
I20260927 21:32:03.991434 21205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb6a341ea8e64267af6d36d1ce9b9616: No bootstrap required, opened a new log
Sep 27 21:32:03 dist-test-slave-bxm0 krb5kdc[21125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 21:32:03.999948 21205 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb6a341ea8e64267af6d36d1ce9b9616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6a341ea8e64267af6d36d1ce9b9616" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } }
I20260927 21:32:04.000425 21205 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb6a341ea8e64267af6d36d1ce9b9616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:32:04.000566 21205 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb6a341ea8e64267af6d36d1ce9b9616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6a341ea8e64267af6d36d1ce9b9616, State: Initialized, Role: FOLLOWER
I20260927 21:32:04.001246 21205 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb6a341ea8e64267af6d36d1ce9b9616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6a341ea8e64267af6d36d1ce9b9616" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } }
I20260927 21:32:04.001521 21205 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb6a341ea8e64267af6d36d1ce9b9616 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:32:04.001698 21205 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb6a341ea8e64267af6d36d1ce9b9616 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:32:04.001884 21205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb6a341ea8e64267af6d36d1ce9b9616 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:32:04.002331 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:03.979571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50035 (local address 127.0.0.1:46481)
0927 21:32:03.980412 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:03.980429 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:03.980471 (+ 42us) server_negotiation.cc:407] Connection header received
0927 21:32:03.982236 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:03.982259 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:03.982657 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:03.983047 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:03.984990 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:03.986331 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:03.988138 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:03.989250 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:03.989401 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:03.994353 (+ 4952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:03.994360 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:03.994376 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:03.994440 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:03.997747 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:03.998511 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:03.998518 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:03.998525 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:03.998704 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:03.999120 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:03.999127 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:03.999138 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:03.999709 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:04.000035 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:04.000522 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 21:32:04.001019 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":384,"threads_started":1}
I20260927 21:32:04.004904 21205 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb6a341ea8e64267af6d36d1ce9b9616 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6a341ea8e64267af6d36d1ce9b9616" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } }
I20260927 21:32:04.005559 21205 leader_election.cc:302] T 00000000000000000000000000000000 P eb6a341ea8e64267af6d36d1ce9b9616 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb6a341ea8e64267af6d36d1ce9b9616; no voters:
I20260927 21:32:04.006227 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:32:04.006300 21205 leader_election.cc:288] T 00000000000000000000000000000000 P eb6a341ea8e64267af6d36d1ce9b9616 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:32:04.006409 21210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb6a341ea8e64267af6d36d1ce9b9616 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:32:04.008855 21205 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb6a341ea8e64267af6d36d1ce9b9616 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:32:04.010176 21210 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb6a341ea8e64267af6d36d1ce9b9616 [term 1 LEADER]: Becoming Leader. State: Replica: eb6a341ea8e64267af6d36d1ce9b9616, State: Running, Role: LEADER
I20260927 21:32:04.011096 21210 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb6a341ea8e64267af6d36d1ce9b9616 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6a341ea8e64267af6d36d1ce9b9616" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } }
I20260927 21:32:04.014672 21211 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6a341ea8e64267af6d36d1ce9b9616 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb6a341ea8e64267af6d36d1ce9b9616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6a341ea8e64267af6d36d1ce9b9616" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } } }
I20260927 21:32:04.015182 21211 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6a341ea8e64267af6d36d1ce9b9616 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:04.015870 21212 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6a341ea8e64267af6d36d1ce9b9616 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb6a341ea8e64267af6d36d1ce9b9616. Latest consensus state: current_term: 1 leader_uuid: "eb6a341ea8e64267af6d36d1ce9b9616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6a341ea8e64267af6d36d1ce9b9616" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } } }
I20260927 21:32:04.016269 21212 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6a341ea8e64267af6d36d1ce9b9616 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:04.023380 21218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:32:04.030681 21218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:32:04.043210 21233 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 21:32:04.043674 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:04.021226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43077 (local address 127.0.0.1:46481)
0927 21:32:04.021405 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:04.021410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:04.022556 (+ 1146us) server_negotiation.cc:407] Connection header received
0927 21:32:04.022901 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:04.022913 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:04.023050 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:04.023152 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:04.025442 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:04.026746 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:04.028785 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:04.029735 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:04.029909 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:04.033989 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:04.033999 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:04.034007 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:04.034062 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:04.038870 (+ 4808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:04.040079 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:04.040087 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:04.040093 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:04.040271 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:04.040783 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:04.040792 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:04.040798 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:04.041368 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:04.041694 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:04.043075 (+ 1381us) server_negotiation.cc:299] Negotiation successful
0927 21:32:04.043321 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:32:04.048178 21218 catalog_manager.cc:1347] Generated new cluster ID: 8efa8e7f6f4f41a38badb1148f24bf15
I20260927 21:32:04.048295 21218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:32:04.060855 21218 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:32:04.062821 21218 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:32:04.072690 21218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb6a341ea8e64267af6d36d1ce9b9616: Generated new TSK 0
I20260927 21:32:04.073736 21218 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:32:04.079327 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 21138
I20260927 21:32:04.081120 21200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:32:04.172353 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 21138
2026-09-27T21:32:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2040 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:32:04 dist-test-slave-bxm0 krb5kdc[21238](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:32:04 dist-test-slave-bxm0 krb5kdc[21238](info): set up 2 sockets
Sep 27 21:32:04 dist-test-slave-bxm0 krb5kdc[21238](info): commencing operation
krb5kdc: starting...
W20260927 21:32:05.298000 388 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:32:05 dist-test-slave-bxm0 krb5kdc[21238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:32:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:32:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:32:05.553041 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45647
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:32:05.889974 21251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:05.890345 21251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:05.890511 21251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:05.897776 21251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:32:05.897946 21251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:05.898058 21251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:05.898164 21251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:32:05.898240 21251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:32:05.908298 21251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:05.909726 21251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:05.911029 21251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:05.922889 21256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:05.923023 21259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:05.922978 21257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:05.924108 21251 server_base.cc:1034] running on GCE node
I20260927 21:32:05.924961 21251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:05.926468 21251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:05.927707 21251 hybrid_clock.cc:648] HybridClock initialized: now 1790544725927680 us; error 48 us; skew 500 ppm
I20260927 21:32:05.928071 21251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:05 dist-test-slave-bxm0 krb5kdc[21238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:05.931205 21251 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:32:05.932752 21251 webserver.cc:460] Webserver started at http://127.0.0.1:43931/ using document root <none> and password file <none>
I20260927 21:32:05.933472 21251 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:05.933688 21251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:05.934026 21251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:05.937041 21251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "e7678302296e4cf2a0d6764a7de7d904"
format_stamp: "Formatted at 2026-09-27 21:32:05 on dist-test-slave-bxm0"
server_key: "4254c7840e23b5006809a7c3edbd53a6"
server_key_iv: "d07be5b95bfe9e49c5af93ca83a7c5cc"
server_key_version: "encryptionkey@0"
I20260927 21:32:05.937912 21251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "e7678302296e4cf2a0d6764a7de7d904"
format_stamp: "Formatted at 2026-09-27 21:32:05 on dist-test-slave-bxm0"
server_key: "4254c7840e23b5006809a7c3edbd53a6"
server_key_iv: "d07be5b95bfe9e49c5af93ca83a7c5cc"
server_key_version: "encryptionkey@0"
I20260927 21:32:05.943228 21251 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 21:32:05.946859 21266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:05.948279 21251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:32:05.948474 21251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "e7678302296e4cf2a0d6764a7de7d904"
format_stamp: "Formatted at 2026-09-27 21:32:05 on dist-test-slave-bxm0"
server_key: "4254c7840e23b5006809a7c3edbd53a6"
server_key_iv: "d07be5b95bfe9e49c5af93ca83a7c5cc"
server_key_version: "encryptionkey@0"
I20260927 21:32:05.948689 21251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:05.968767 21251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:05.972972 21251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:05.973292 21251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:05.992470 21251 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45647
I20260927 21:32:05.992501 21317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45647 every 8 connection(s)
I20260927 21:32:05.994006 21251 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:32:05.997761 21318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:32:05.998428 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 21251
I20260927 21:32:05.998624 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:32:05.999190 388 external_mini_cluster.cc:1428] Setting key 687eedae24099f2a42238de9c797798c
I20260927 21:32:06.011390 21318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7678302296e4cf2a0d6764a7de7d904: Bootstrap starting.
I20260927 21:32:06.012993 21321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:06.000971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54977 (local address 127.0.0.1:45647)
0927 21:32:06.001837 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:06.001849 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:06.001889 (+ 40us) server_negotiation.cc:407] Connection header received
0927 21:32:06.004146 (+ 2257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:06.004170 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:06.004605 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:06.004945 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:06.005553 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:06.006870 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:06.008325 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:06.009077 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:06.009356 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:06.010644 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:06.010652 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:06.010668 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:32:06.011231 (+ 563us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:06.011315 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:32:06.011341 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:06.011486 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:06.011698 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 21:32:06.011992 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":395,"threads_started":1}
I20260927 21:32:06.015070 21318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7678302296e4cf2a0d6764a7de7d904: Neither blocks nor log segments found. Creating new log.
I20260927 21:32:06.016599 21318 log.cc:824] T 00000000000000000000000000000000 P e7678302296e4cf2a0d6764a7de7d904: Log is configured to *not* fsync() on all Append() calls
I20260927 21:32:06.017117 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:32:06.019661 21318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7678302296e4cf2a0d6764a7de7d904: No bootstrap required, opened a new log
I20260927 21:32:06.028573 21321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:06.021746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57965 (local address 127.0.0.1:45647)
0927 21:32:06.021975 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:06.021981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:06.022237 (+ 256us) server_negotiation.cc:407] Connection header received
0927 21:32:06.022405 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:06.022412 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:06.022477 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:06.022580 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:06.023151 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:06.023874 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:06.025899 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:06.026384 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:06.026491 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:06.027782 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:06.027786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:06.027793 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:32:06.027813 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:06.027857 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:32:06.027875 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:06.027945 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:06.028107 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 21:32:06.028188 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 21:32:06.030292 21318 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7678302296e4cf2a0d6764a7de7d904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7678302296e4cf2a0d6764a7de7d904" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45647 } }
I20260927 21:32:06.030733 21318 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7678302296e4cf2a0d6764a7de7d904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:32:06.030843 21318 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7678302296e4cf2a0d6764a7de7d904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7678302296e4cf2a0d6764a7de7d904, State: Initialized, Role: FOLLOWER
I20260927 21:32:06.031422 21318 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7678302296e4cf2a0d6764a7de7d904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7678302296e4cf2a0d6764a7de7d904" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45647 } }
I20260927 21:32:06.031685 21318 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7678302296e4cf2a0d6764a7de7d904 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:32:06.031863 21318 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7678302296e4cf2a0d6764a7de7d904 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:32:06.032052 21318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7678302296e4cf2a0d6764a7de7d904 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:32:06.034399 21318 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7678302296e4cf2a0d6764a7de7d904 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7678302296e4cf2a0d6764a7de7d904" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45647 } }
I20260927 21:32:06.034922 21318 leader_election.cc:302] T 00000000000000000000000000000000 P e7678302296e4cf2a0d6764a7de7d904 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7678302296e4cf2a0d6764a7de7d904; no voters:
I20260927 21:32:06.035583 21318 leader_election.cc:288] T 00000000000000000000000000000000 P e7678302296e4cf2a0d6764a7de7d904 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:32:06.035708 21328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7678302296e4cf2a0d6764a7de7d904 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:32:06.037091 21328 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7678302296e4cf2a0d6764a7de7d904 [term 1 LEADER]: Becoming Leader. State: Replica: e7678302296e4cf2a0d6764a7de7d904, State: Running, Role: LEADER
I20260927 21:32:06.037734 21318 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7678302296e4cf2a0d6764a7de7d904 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:32:06.037707 21328 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7678302296e4cf2a0d6764a7de7d904 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7678302296e4cf2a0d6764a7de7d904" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45647 } }
I20260927 21:32:06.040488 21330 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7678302296e4cf2a0d6764a7de7d904 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7678302296e4cf2a0d6764a7de7d904. Latest consensus state: current_term: 1 leader_uuid: "e7678302296e4cf2a0d6764a7de7d904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7678302296e4cf2a0d6764a7de7d904" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45647 } } }
I20260927 21:32:06.040804 21330 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7678302296e4cf2a0d6764a7de7d904 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:06.042387 21336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:32:06.041414 21329 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7678302296e4cf2a0d6764a7de7d904 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7678302296e4cf2a0d6764a7de7d904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7678302296e4cf2a0d6764a7de7d904" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45647 } } }
I20260927 21:32:06.043109 21329 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7678302296e4cf2a0d6764a7de7d904 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:06.048697 21336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:32:06.058965 21336 catalog_manager.cc:1347] Generated new cluster ID: f2d5fae17cc9430a8d2039a886cac06c
I20260927 21:32:06.059084 21336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:32:06.081514 21336 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:32:06.082962 21336 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:32:06.095316 21336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7678302296e4cf2a0d6764a7de7d904: Generated new TSK 0
I20260927 21:32:06.096386 21336 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:32:06.115265 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 21251
W20260927 21:32:06.116503 21313 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54977
W20260927 21:32:06.117359 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 21251: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:32:06.117516 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 21251
2026-09-27T21:32:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1941 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:32:06 dist-test-slave-bxm0 krb5kdc[21349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:32:06 dist-test-slave-bxm0 krb5kdc[21349](info): set up 2 sockets
Sep 27 21:32:06 dist-test-slave-bxm0 krb5kdc[21349](info): commencing operation
krb5kdc: starting...
W20260927 21:32:07.221988 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:32:07 dist-test-slave-bxm0 krb5kdc[21349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:32:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:32:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:32:07.481920 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:32:07.822005 21362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:07.822317 21362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:07.822412 21362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:07.829846 21362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:32:07.829964 21362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:07.830009 21362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:07.830066 21362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:32:07.830111 21362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:32:07.840191 21362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:07.841593 21362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:07.842913 21362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:07.855149 21367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:07.855100 21370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:07.855121 21368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:07.856518 21362 server_base.cc:1034] running on GCE node
I20260927 21:32:07.857283 21362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:07.858736 21362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:07.859942 21362 hybrid_clock.cc:648] HybridClock initialized: now 1790544727859910 us; error 27 us; skew 500 ppm
I20260927 21:32:07.860275 21362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:07 dist-test-slave-bxm0 krb5kdc[21349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:07.863394 21362 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:32:07.865443 21362 webserver.cc:460] Webserver started at http://127.0.0.1:39447/ using document root <none> and password file <none>
I20260927 21:32:07.866205 21362 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:07.866349 21362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:07.866673 21362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:07.869725 21362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "550ec0852d5b4381abc31389d7bce4af"
format_stamp: "Formatted at 2026-09-27 21:32:07 on dist-test-slave-bxm0"
server_key: "7b7b8d0b34e7a732ca3ecc4615bd09de"
server_key_iv: "fbc7d6e7736dfbfc71244532a1c44c80"
server_key_version: "encryptionkey@0"
I20260927 21:32:07.870491 21362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "550ec0852d5b4381abc31389d7bce4af"
format_stamp: "Formatted at 2026-09-27 21:32:07 on dist-test-slave-bxm0"
server_key: "7b7b8d0b34e7a732ca3ecc4615bd09de"
server_key_iv: "fbc7d6e7736dfbfc71244532a1c44c80"
server_key_version: "encryptionkey@0"
I20260927 21:32:07.875695 21362 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:32:07.879112 21377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:07.880424 21362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:32:07.880609 21362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "550ec0852d5b4381abc31389d7bce4af"
format_stamp: "Formatted at 2026-09-27 21:32:07 on dist-test-slave-bxm0"
server_key: "7b7b8d0b34e7a732ca3ecc4615bd09de"
server_key_iv: "fbc7d6e7736dfbfc71244532a1c44c80"
server_key_version: "encryptionkey@0"
I20260927 21:32:07.880815 21362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:07.886620 21362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:07.890774 21362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:07.891062 21362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:07.910171 21362 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44815
I20260927 21:32:07.910220 21428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44815 every 8 connection(s)
I20260927 21:32:07.911588 21362 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:32:07.915657 21429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:32:07.917177 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 21362
I20260927 21:32:07.917359 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:32:07.917958 388 external_mini_cluster.cc:1428] Setting key 5151a7211ecd8d18e014e66c3f9723f4
I20260927 21:32:07.925951 21432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:07.919933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42311 (local address 127.0.0.1:44815)
0927 21:32:07.920804 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:07.920819 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:07.920862 (+ 43us) server_negotiation.cc:407] Connection header received
0927 21:32:07.922541 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:07.922565 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:07.922929 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:07.923282 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:07.924361 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:07.924367 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:07.924379 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:32:07.924392 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:07.924446 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:32:07.924470 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:07.924630 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:07.924751 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 21:32:07.924952 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":420,"threads_started":1}
I20260927 21:32:07.929546 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:32:07.929571 21429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 550ec0852d5b4381abc31389d7bce4af: Bootstrap starting.
I20260927 21:32:07.932615 21429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 550ec0852d5b4381abc31389d7bce4af: Neither blocks nor log segments found. Creating new log.
I20260927 21:32:07.933736 21429 log.cc:824] T 00000000000000000000000000000000 P 550ec0852d5b4381abc31389d7bce4af: Log is configured to *not* fsync() on all Append() calls
I20260927 21:32:07.936690 21429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 550ec0852d5b4381abc31389d7bce4af: No bootstrap required, opened a new log
I20260927 21:32:07.936899 21432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:07.933948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60911 (local address 127.0.0.1:44815)
0927 21:32:07.934161 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:07.934168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:07.934576 (+ 408us) server_negotiation.cc:407] Connection header received
0927 21:32:07.934790 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:07.934799 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:07.934847 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:07.934952 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:07.936117 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:07.936126 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:07.936135 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:32:07.936153 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:07.936194 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:32:07.936212 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:07.936346 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:07.936465 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 21:32:07.936531 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 21:32:07.945307 21429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 550ec0852d5b4381abc31389d7bce4af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "550ec0852d5b4381abc31389d7bce4af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44815 } }
I20260927 21:32:07.945755 21429 raft_consensus.cc:374] T 00000000000000000000000000000000 P 550ec0852d5b4381abc31389d7bce4af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:32:07.945878 21429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 550ec0852d5b4381abc31389d7bce4af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 550ec0852d5b4381abc31389d7bce4af, State: Initialized, Role: FOLLOWER
I20260927 21:32:07.946502 21429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 550ec0852d5b4381abc31389d7bce4af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "550ec0852d5b4381abc31389d7bce4af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44815 } }
I20260927 21:32:07.946734 21429 raft_consensus.cc:388] T 00000000000000000000000000000000 P 550ec0852d5b4381abc31389d7bce4af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:32:07.946890 21429 raft_consensus.cc:482] T 00000000000000000000000000000000 P 550ec0852d5b4381abc31389d7bce4af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:32:07.947068 21429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 550ec0852d5b4381abc31389d7bce4af [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:32:07.949255 21429 raft_consensus.cc:504] T 00000000000000000000000000000000 P 550ec0852d5b4381abc31389d7bce4af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "550ec0852d5b4381abc31389d7bce4af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44815 } }
I20260927 21:32:07.949834 21429 leader_election.cc:302] T 00000000000000000000000000000000 P 550ec0852d5b4381abc31389d7bce4af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 550ec0852d5b4381abc31389d7bce4af; no voters:
I20260927 21:32:07.950500 21429 leader_election.cc:288] T 00000000000000000000000000000000 P 550ec0852d5b4381abc31389d7bce4af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:32:07.950729 21439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 550ec0852d5b4381abc31389d7bce4af [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:32:07.951483 21439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 550ec0852d5b4381abc31389d7bce4af [term 1 LEADER]: Becoming Leader. State: Replica: 550ec0852d5b4381abc31389d7bce4af, State: Running, Role: LEADER
I20260927 21:32:07.952099 21439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 550ec0852d5b4381abc31389d7bce4af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "550ec0852d5b4381abc31389d7bce4af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44815 } }
I20260927 21:32:07.952672 21429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 550ec0852d5b4381abc31389d7bce4af [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:32:07.955638 21441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 550ec0852d5b4381abc31389d7bce4af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 550ec0852d5b4381abc31389d7bce4af. Latest consensus state: current_term: 1 leader_uuid: "550ec0852d5b4381abc31389d7bce4af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "550ec0852d5b4381abc31389d7bce4af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44815 } } }
I20260927 21:32:07.955613 21440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 550ec0852d5b4381abc31389d7bce4af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "550ec0852d5b4381abc31389d7bce4af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "550ec0852d5b4381abc31389d7bce4af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44815 } } }
I20260927 21:32:07.956130 21441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 550ec0852d5b4381abc31389d7bce4af [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:07.956172 21440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 550ec0852d5b4381abc31389d7bce4af [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:07.958810 21447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:32:07.963979 21447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:32:07.973053 21447 catalog_manager.cc:1347] Generated new cluster ID: 37f16c990246440391ae1b572a89ac9d
I20260927 21:32:07.973186 21447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:32:08.000108 21447 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:32:08.000699 21447 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:32:08.014874 21447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 550ec0852d5b4381abc31389d7bce4af: Generated new TSK 0
I20260927 21:32:08.016088 21447 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:32:08.021552 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 21362
W20260927 21:32:08.022624 21424 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42311
W20260927 21:32:08.023370 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 21362: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:32:08.023528 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 21362
2026-09-27T21:32:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1905 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:32:08 dist-test-slave-bxm0 krb5kdc[21460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:32:08 dist-test-slave-bxm0 krb5kdc[21460](info): set up 2 sockets
Sep 27 21:32:08 dist-test-slave-bxm0 krb5kdc[21460](info): commencing operation
krb5kdc: starting...
W20260927 21:32:09.134106 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:32:09 dist-test-slave-bxm0 krb5kdc[21460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:32:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:32:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:32:09.392673 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40569
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:32:09.736989 21473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:09.737357 21473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:09.737520 21473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:09.744891 21473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:32:09.745076 21473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:09.745193 21473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:09.745306 21473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:32:09.745395 21473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:32:09.755534 21473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:09.756897 21473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:09.758262 21473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:09.770026 21478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:09.770066 21479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:09.770107 21481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:09.771287 21473 server_base.cc:1034] running on GCE node
I20260927 21:32:09.772190 21473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:09.773600 21473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:09.774869 21473 hybrid_clock.cc:648] HybridClock initialized: now 1790544729774830 us; error 34 us; skew 500 ppm
I20260927 21:32:09.775233 21473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:09 dist-test-slave-bxm0 krb5kdc[21460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:09.782377 21473 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:32:09.784083 21473 webserver.cc:460] Webserver started at http://127.0.0.1:37477/ using document root <none> and password file <none>
I20260927 21:32:09.784878 21473 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:09.785035 21473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:09.785363 21473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:09.788486 21473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "dfa37c7d356e4147bbc808a94adfc694"
format_stamp: "Formatted at 2026-09-27 21:32:09 on dist-test-slave-bxm0"
server_key: "aa35c80a2993f2dd58f7c086e715fadb"
server_key_iv: "c1407f0e9bf65d99a173ca77a950e021"
server_key_version: "encryptionkey@0"
I20260927 21:32:09.789259 21473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "dfa37c7d356e4147bbc808a94adfc694"
format_stamp: "Formatted at 2026-09-27 21:32:09 on dist-test-slave-bxm0"
server_key: "aa35c80a2993f2dd58f7c086e715fadb"
server_key_iv: "c1407f0e9bf65d99a173ca77a950e021"
server_key_version: "encryptionkey@0"
I20260927 21:32:09.794720 21473 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 21:32:09.798386 21488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:09.799813 21473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:32:09.800011 21473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "dfa37c7d356e4147bbc808a94adfc694"
format_stamp: "Formatted at 2026-09-27 21:32:09 on dist-test-slave-bxm0"
server_key: "aa35c80a2993f2dd58f7c086e715fadb"
server_key_iv: "c1407f0e9bf65d99a173ca77a950e021"
server_key_version: "encryptionkey@0"
I20260927 21:32:09.800222 21473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:09.811419 21473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:09.815408 21473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:09.815711 21473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:09.834769 21473 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40569
I20260927 21:32:09.834811 21539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40569 every 8 connection(s)
I20260927 21:32:09.836230 21473 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:32:09.838073 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 21473
I20260927 21:32:09.838267 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:32:09.838809 388 external_mini_cluster.cc:1428] Setting key 801fe22003b9d8f772ddeaaccd3fd0f1
I20260927 21:32:09.841951 21540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 21:32:09 dist-test-slave-bxm0 krb5kdc[21460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 21:32:09.852517 21540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfa37c7d356e4147bbc808a94adfc694: Bootstrap starting.
I20260927 21:32:09.855963 21540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfa37c7d356e4147bbc808a94adfc694: Neither blocks nor log segments found. Creating new log.
I20260927 21:32:09.856983 21540 log.cc:824] T 00000000000000000000000000000000 P dfa37c7d356e4147bbc808a94adfc694: Log is configured to *not* fsync() on all Append() calls
I20260927 21:32:09.859953 21540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfa37c7d356e4147bbc808a94adfc694: No bootstrap required, opened a new log
I20260927 21:32:09.860280 21543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:09.840453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:42297 (local address 127.0.0.1:40569)
0927 21:32:09.841093 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:09.841105 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:09.841145 (+ 40us) server_negotiation.cc:407] Connection header received
0927 21:32:09.842885 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:09.842907 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:09.843301 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:09.843683 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:09.845126 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:09.846545 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:09.847835 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:09.848522 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:09.848712 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:09.852899 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:09.852907 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:09.852921 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:09.852961 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:09.855970 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:09.856864 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:09.856870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:09.856877 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:09.857033 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:09.857523 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:09.857527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:09.857529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:09.858107 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:09.858455 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:09.858926 (+ 471us) server_negotiation.cc:299] Negotiation successful
0927 21:32:09.859248 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":264,"threads_started":1}
I20260927 21:32:09.864238 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:32:09.870004 21540 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfa37c7d356e4147bbc808a94adfc694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa37c7d356e4147bbc808a94adfc694" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40569 } }
I20260927 21:32:09.870527 21540 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfa37c7d356e4147bbc808a94adfc694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:32:09.870669 21540 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfa37c7d356e4147bbc808a94adfc694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa37c7d356e4147bbc808a94adfc694, State: Initialized, Role: FOLLOWER
I20260927 21:32:09.871371 21540 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfa37c7d356e4147bbc808a94adfc694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa37c7d356e4147bbc808a94adfc694" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40569 } }
I20260927 21:32:09.871675 21540 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfa37c7d356e4147bbc808a94adfc694 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:32:09.871829 21540 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfa37c7d356e4147bbc808a94adfc694 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:32:09.872020 21540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfa37c7d356e4147bbc808a94adfc694 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:32:09.875041 21540 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfa37c7d356e4147bbc808a94adfc694 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa37c7d356e4147bbc808a94adfc694" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40569 } }
I20260927 21:32:09.875646 21540 leader_election.cc:302] T 00000000000000000000000000000000 P dfa37c7d356e4147bbc808a94adfc694 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfa37c7d356e4147bbc808a94adfc694; no voters:
I20260927 21:32:09.876345 21540 leader_election.cc:288] T 00000000000000000000000000000000 P dfa37c7d356e4147bbc808a94adfc694 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:32:09.876466 21550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfa37c7d356e4147bbc808a94adfc694 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:32:09.877713 21550 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfa37c7d356e4147bbc808a94adfc694 [term 1 LEADER]: Becoming Leader. State: Replica: dfa37c7d356e4147bbc808a94adfc694, State: Running, Role: LEADER
I20260927 21:32:09.878332 21550 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfa37c7d356e4147bbc808a94adfc694 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa37c7d356e4147bbc808a94adfc694" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40569 } }
I20260927 21:32:09.879313 21540 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfa37c7d356e4147bbc808a94adfc694 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:32:09.881811 21552 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfa37c7d356e4147bbc808a94adfc694 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfa37c7d356e4147bbc808a94adfc694. Latest consensus state: current_term: 1 leader_uuid: "dfa37c7d356e4147bbc808a94adfc694" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa37c7d356e4147bbc808a94adfc694" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40569 } } }
I20260927 21:32:09.882123 21552 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfa37c7d356e4147bbc808a94adfc694 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:09.881350 21551 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfa37c7d356e4147bbc808a94adfc694 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfa37c7d356e4147bbc808a94adfc694" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa37c7d356e4147bbc808a94adfc694" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40569 } } }
I20260927 21:32:09.882606 21551 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfa37c7d356e4147bbc808a94adfc694 [sys.catalog]: This master's current role is: LEADER
W20260927 21:32:09.890103 21566 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dfa37c7d356e4147bbc808a94adfc694: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:32:09.890341 21566 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:32:09.890523 21558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:32:09.892495 21543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:09.869750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:59589 (local address 127.0.0.1:40569)
0927 21:32:09.869938 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:09.869945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:09.870487 (+ 542us) server_negotiation.cc:407] Connection header received
0927 21:32:09.870720 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:09.870724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:09.870817 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:09.870908 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:09.872774 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:09.873693 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:09.875507 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:09.876038 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:09.876190 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:09.885126 (+ 8936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:09.885135 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:09.885143 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:09.885211 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:09.888929 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:09.890056 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:09.890063 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:09.890068 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:09.890287 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:09.890860 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:09.890864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:09.890866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:09.891178 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:09.891405 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:09.891919 (+ 514us) server_negotiation.cc:299] Negotiation successful
0927 21:32:09.892113 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:32:09.892714 21558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:32:09.901129 21558 catalog_manager.cc:1347] Generated new cluster ID: 958f85ecd37c462599428cd754b60b62
I20260927 21:32:09.901280 21558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:32:09.914364 21558 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:32:09.916006 21558 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:32:09.923964 21558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfa37c7d356e4147bbc808a94adfc694: Generated new TSK 0
I20260927 21:32:09.925045 21558 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:32:09.938565 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 21473
I20260927 21:32:09.939800 21535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:32:10.031423 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 21473
W20260927 21:32:10.053844 21458 connection.cc:535] client connection to 127.0.0.1:40569 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40569): Connection reset by peer (error 104)
2026-09-27T21:32:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2014 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:32:10 dist-test-slave-bxm0 krb5kdc[21572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:32:10 dist-test-slave-bxm0 krb5kdc[21572](info): set up 2 sockets
Sep 27 21:32:10 dist-test-slave-bxm0 krb5kdc[21572](info): commencing operation
krb5kdc: starting...
W20260927 21:32:11.150075 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:32:11 dist-test-slave-bxm0 krb5kdc[21572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:32:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:32:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:32:11.416347 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35905
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:32:11.754664 21585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:11.754938 21585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:11.755136 21585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:11.762385 21585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:32:11.762532 21585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:11.762641 21585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:11.762753 21585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:32:11.762838 21585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:32:11.773772 21585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:11.775228 21585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:11.776712 21585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:11.788980 21591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:11.789055 21593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:11.789033 21590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:11.789958 21585 server_base.cc:1034] running on GCE node
I20260927 21:32:11.790735 21585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:11.792143 21585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:11.793360 21585 hybrid_clock.cc:648] HybridClock initialized: now 1790544731793315 us; error 40 us; skew 500 ppm
I20260927 21:32:11.793860 21585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:11 dist-test-slave-bxm0 krb5kdc[21572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:11.797695 21585 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:32:11.799314 21585 webserver.cc:460] Webserver started at http://127.0.0.1:44089/ using document root <none> and password file <none>
I20260927 21:32:11.800047 21585 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:11.800207 21585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:11.800555 21585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:11.803766 21585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "4f9cb051be5a4ff49d8e2bb99536abc9"
format_stamp: "Formatted at 2026-09-27 21:32:11 on dist-test-slave-bxm0"
server_key: "53088975f31b0fcb06f727ce57a3fda1"
server_key_iv: "2b1bc5da2b6b6230b52fca95ec36309b"
server_key_version: "encryptionkey@0"
I20260927 21:32:11.804565 21585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "4f9cb051be5a4ff49d8e2bb99536abc9"
format_stamp: "Formatted at 2026-09-27 21:32:11 on dist-test-slave-bxm0"
server_key: "53088975f31b0fcb06f727ce57a3fda1"
server_key_iv: "2b1bc5da2b6b6230b52fca95ec36309b"
server_key_version: "encryptionkey@0"
I20260927 21:32:11.810143 21585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 21:32:11.813722 21600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:11.815086 21585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:32:11.815287 21585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "4f9cb051be5a4ff49d8e2bb99536abc9"
format_stamp: "Formatted at 2026-09-27 21:32:11 on dist-test-slave-bxm0"
server_key: "53088975f31b0fcb06f727ce57a3fda1"
server_key_iv: "2b1bc5da2b6b6230b52fca95ec36309b"
server_key_version: "encryptionkey@0"
I20260927 21:32:11.815519 21585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:11.836623 21585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:11.841006 21585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:11.841325 21585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:11.861598 21585 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35905
I20260927 21:32:11.861637 21651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35905 every 8 connection(s)
I20260927 21:32:11.863256 21585 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:32:11.867223 21652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:32:11.872519 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 21585
I20260927 21:32:11.872742 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:32:11.873191 388 external_mini_cluster.cc:1428] Setting key 7922a35fd93125e12cdd0de47d89d78b
I20260927 21:32:11.878342 21652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f9cb051be5a4ff49d8e2bb99536abc9: Bootstrap starting.
I20260927 21:32:11.882185 21652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f9cb051be5a4ff49d8e2bb99536abc9: Neither blocks nor log segments found. Creating new log.
I20260927 21:32:11.883417 21652 log.cc:824] T 00000000000000000000000000000000 P 4f9cb051be5a4ff49d8e2bb99536abc9: Log is configured to *not* fsync() on all Append() calls
I20260927 21:32:11.886780 21652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f9cb051be5a4ff49d8e2bb99536abc9: No bootstrap required, opened a new log
I20260927 21:32:11.887028 21655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:11.875178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:50611 (local address 127.0.0.1:35905)
0927 21:32:11.876054 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:11.876067 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:11.876107 (+ 40us) server_negotiation.cc:407] Connection header received
0927 21:32:11.877841 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:11.877864 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:11.878275 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:11.878656 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:11.879237 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:11.880558 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:11.882502 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:11.883186 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:11.883327 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:11.884721 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:11.884731 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:11.884747 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:32:11.885304 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:11.885380 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:32:11.885405 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:11.885572 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:11.885893 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 21:32:11.886070 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":405,"threads_started":1}
I20260927 21:32:11.891211 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:32:11.895365 21652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f9cb051be5a4ff49d8e2bb99536abc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f9cb051be5a4ff49d8e2bb99536abc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } }
I20260927 21:32:11.895905 21652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f9cb051be5a4ff49d8e2bb99536abc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:32:11.896073 21652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f9cb051be5a4ff49d8e2bb99536abc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f9cb051be5a4ff49d8e2bb99536abc9, State: Initialized, Role: FOLLOWER
I20260927 21:32:11.897493 21652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f9cb051be5a4ff49d8e2bb99536abc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f9cb051be5a4ff49d8e2bb99536abc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } }
I20260927 21:32:11.897883 21652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f9cb051be5a4ff49d8e2bb99536abc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:32:11.898083 21652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f9cb051be5a4ff49d8e2bb99536abc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:32:11.898335 21652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f9cb051be5a4ff49d8e2bb99536abc9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:32:11.901356 21652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f9cb051be5a4ff49d8e2bb99536abc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f9cb051be5a4ff49d8e2bb99536abc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } }
I20260927 21:32:11.902053 21652 leader_election.cc:302] T 00000000000000000000000000000000 P 4f9cb051be5a4ff49d8e2bb99536abc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f9cb051be5a4ff49d8e2bb99536abc9; no voters:
I20260927 21:32:11.902855 21652 leader_election.cc:288] T 00000000000000000000000000000000 P 4f9cb051be5a4ff49d8e2bb99536abc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:32:11.903010 21662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f9cb051be5a4ff49d8e2bb99536abc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:32:11.903393 21655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:11.895616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:33863 (local address 127.0.0.1:35905)
0927 21:32:11.895817 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:11.895823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:11.896773 (+ 950us) server_negotiation.cc:407] Connection header received
0927 21:32:11.896911 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:11.896918 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:11.897007 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:11.897124 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:11.897712 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:11.898959 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:11.900302 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:11.901057 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:11.901216 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:11.902424 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:11.902429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:11.902432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:32:11.902450 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:11.902488 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:32:11.902504 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:11.902634 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:11.902853 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 21:32:11.902913 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 21:32:11.905898 21652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f9cb051be5a4ff49d8e2bb99536abc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:32:11.905898 21662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f9cb051be5a4ff49d8e2bb99536abc9 [term 1 LEADER]: Becoming Leader. State: Replica: 4f9cb051be5a4ff49d8e2bb99536abc9, State: Running, Role: LEADER
I20260927 21:32:11.906605 21662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f9cb051be5a4ff49d8e2bb99536abc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f9cb051be5a4ff49d8e2bb99536abc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } }
I20260927 21:32:11.910156 21663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f9cb051be5a4ff49d8e2bb99536abc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f9cb051be5a4ff49d8e2bb99536abc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f9cb051be5a4ff49d8e2bb99536abc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } } }
I20260927 21:32:11.910590 21663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f9cb051be5a4ff49d8e2bb99536abc9 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:11.910101 21664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f9cb051be5a4ff49d8e2bb99536abc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f9cb051be5a4ff49d8e2bb99536abc9. Latest consensus state: current_term: 1 leader_uuid: "4f9cb051be5a4ff49d8e2bb99536abc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f9cb051be5a4ff49d8e2bb99536abc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35905 } } }
I20260927 21:32:11.911146 21664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f9cb051be5a4ff49d8e2bb99536abc9 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:11.911921 21675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:32:11.918561 21675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:32:11.927910 21675 catalog_manager.cc:1347] Generated new cluster ID: 64bcbaeb610d4f34ae14158d740fa4ff
I20260927 21:32:11.928069 21675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:32:11.956718 21675 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:32:11.958623 21675 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:32:11.965778 21675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f9cb051be5a4ff49d8e2bb99536abc9: Generated new TSK 0
I20260927 21:32:11.966602 21675 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:32:11.976250 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 21585
W20260927 21:32:11.977528 21647 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.24:50611
W20260927 21:32:11.978361 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 21585: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:32:11.978505 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 21585
W20260927 21:32:11.995560 21570 connection.cc:535] client connection to 127.0.0.1:35905 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35905): Connection reset by peer (error 104)
2026-09-27T21:32:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1940 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:32:12 dist-test-slave-bxm0 krb5kdc[21683](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:32:12 dist-test-slave-bxm0 krb5kdc[21683](info): set up 2 sockets
Sep 27 21:32:12 dist-test-slave-bxm0 krb5kdc[21683](info): commencing operation
krb5kdc: starting...
W20260927 21:32:13.078226 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:32:13 dist-test-slave-bxm0 krb5kdc[21683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:32:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:32:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:32:13.338579 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38571
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:32:13.677898 21696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:13.678252 21696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:13.678426 21696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:13.686019 21696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:32:13.686137 21696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:13.686224 21696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:13.686345 21696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:32:13.686424 21696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:32:13.696667 21696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:13.698058 21696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:13.699411 21696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:13.716041 21701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:13.716073 21702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:13.716656 21704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:13.717855 21696 server_base.cc:1034] running on GCE node
I20260927 21:32:13.718713 21696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:13.720264 21696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:13.721539 21696 hybrid_clock.cc:648] HybridClock initialized: now 1790544733721490 us; error 59 us; skew 500 ppm
I20260927 21:32:13.721962 21696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:13 dist-test-slave-bxm0 krb5kdc[21683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:13.725430 21696 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:32:13.727010 21696 webserver.cc:460] Webserver started at http://127.0.0.1:43291/ using document root <none> and password file <none>
I20260927 21:32:13.727837 21696 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:13.727991 21696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:13.728307 21696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:13.731408 21696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "838731ced87148909449badcd3486e59"
format_stamp: "Formatted at 2026-09-27 21:32:13 on dist-test-slave-bxm0"
server_key: "0afce9907b2cb4760937b2225e211bf1"
server_key_iv: "718189e2e3e2f1a45683cca73ab5850d"
server_key_version: "encryptionkey@0"
I20260927 21:32:13.732184 21696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "838731ced87148909449badcd3486e59"
format_stamp: "Formatted at 2026-09-27 21:32:13 on dist-test-slave-bxm0"
server_key: "0afce9907b2cb4760937b2225e211bf1"
server_key_iv: "718189e2e3e2f1a45683cca73ab5850d"
server_key_version: "encryptionkey@0"
I20260927 21:32:13.737543 21696 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 21:32:13.740960 21711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:13.742323 21696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 21:32:13.742522 21696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "838731ced87148909449badcd3486e59"
format_stamp: "Formatted at 2026-09-27 21:32:13 on dist-test-slave-bxm0"
server_key: "0afce9907b2cb4760937b2225e211bf1"
server_key_iv: "718189e2e3e2f1a45683cca73ab5850d"
server_key_version: "encryptionkey@0"
I20260927 21:32:13.742718 21696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:13.748425 21696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:13.752880 21696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:13.753180 21696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:13.771579 21696 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38571
I20260927 21:32:13.771620 21762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38571 every 8 connection(s)
I20260927 21:32:13.773054 21696 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:32:13.773417 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 21696
I20260927 21:32:13.773602 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:32:13.774278 388 external_mini_cluster.cc:1428] Setting key 20d6c3ba51069e5c231d9808740b31db
I20260927 21:32:13.778133 21763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:32:13.783380 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:13.776124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:47397 (local address 127.0.0.1:38571)
0927 21:32:13.776953 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:13.776968 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:13.777020 (+ 52us) server_negotiation.cc:407] Connection header received
0927 21:32:13.779455 (+ 2435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:13.779483 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:13.779931 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:13.780379 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:13.781378 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:13.781387 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:13.781408 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:32:13.781426 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:13.781515 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:32:13.781544 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:13.781782 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:13.781939 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 21:32:13.782134 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":331,"threads_started":1}
I20260927 21:32:13.787082 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:32:13.791083 21763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 838731ced87148909449badcd3486e59: Bootstrap starting.
I20260927 21:32:13.794137 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:13.791566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:43489 (local address 127.0.0.1:38571)
0927 21:32:13.791771 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:13.791779 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:13.792123 (+ 344us) server_negotiation.cc:407] Connection header received
0927 21:32:13.792317 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:13.792323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:13.792383 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:13.792487 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:13.793412 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:13.793418 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:13.793423 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:32:13.793441 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:13.793490 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:32:13.793506 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:13.793619 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:13.793790 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 21:32:13.793859 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 21:32:13.794790 21763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 838731ced87148909449badcd3486e59: Neither blocks nor log segments found. Creating new log.
I20260927 21:32:13.795953 21763 log.cc:824] T 00000000000000000000000000000000 P 838731ced87148909449badcd3486e59: Log is configured to *not* fsync() on all Append() calls
I20260927 21:32:13.798620 21763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 838731ced87148909449badcd3486e59: No bootstrap required, opened a new log
I20260927 21:32:13.806118 21763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 838731ced87148909449badcd3486e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838731ced87148909449badcd3486e59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38571 } }
I20260927 21:32:13.806489 21763 raft_consensus.cc:374] T 00000000000000000000000000000000 P 838731ced87148909449badcd3486e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:32:13.806607 21763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 838731ced87148909449badcd3486e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 838731ced87148909449badcd3486e59, State: Initialized, Role: FOLLOWER
I20260927 21:32:13.807227 21763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 838731ced87148909449badcd3486e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838731ced87148909449badcd3486e59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38571 } }
I20260927 21:32:13.807453 21763 raft_consensus.cc:388] T 00000000000000000000000000000000 P 838731ced87148909449badcd3486e59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:32:13.807621 21763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 838731ced87148909449badcd3486e59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:32:13.807796 21763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 838731ced87148909449badcd3486e59 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:32:13.810173 21763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 838731ced87148909449badcd3486e59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838731ced87148909449badcd3486e59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38571 } }
I20260927 21:32:13.810669 21763 leader_election.cc:302] T 00000000000000000000000000000000 P 838731ced87148909449badcd3486e59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 838731ced87148909449badcd3486e59; no voters:
I20260927 21:32:13.811244 21763 leader_election.cc:288] T 00000000000000000000000000000000 P 838731ced87148909449badcd3486e59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:32:13.811389 21773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 838731ced87148909449badcd3486e59 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:32:13.812142 21773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 838731ced87148909449badcd3486e59 [term 1 LEADER]: Becoming Leader. State: Replica: 838731ced87148909449badcd3486e59, State: Running, Role: LEADER
I20260927 21:32:13.812618 21773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 838731ced87148909449badcd3486e59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838731ced87148909449badcd3486e59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38571 } }
I20260927 21:32:13.813390 21763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 838731ced87148909449badcd3486e59 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:32:13.815907 21775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 838731ced87148909449badcd3486e59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 838731ced87148909449badcd3486e59. Latest consensus state: current_term: 1 leader_uuid: "838731ced87148909449badcd3486e59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838731ced87148909449badcd3486e59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38571 } } }
I20260927 21:32:13.815757 21774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 838731ced87148909449badcd3486e59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "838731ced87148909449badcd3486e59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838731ced87148909449badcd3486e59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38571 } } }
I20260927 21:32:13.816346 21775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 838731ced87148909449badcd3486e59 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:13.816397 21774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 838731ced87148909449badcd3486e59 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:13.822604 21781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:32:13.826809 21781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:32:13.834961 21781 catalog_manager.cc:1347] Generated new cluster ID: 385418ec63514ea4ae6a7a82885b8468
I20260927 21:32:13.835085 21781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:32:13.858767 21781 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:32:13.859277 21781 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:32:13.869055 21781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 838731ced87148909449badcd3486e59: Generated new TSK 0
I20260927 21:32:13.870211 21781 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:32:13.886721 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 21696
W20260927 21:32:13.887832 21758 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.24:47397
W20260927 21:32:13.888582 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 21696: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:32:13.888734 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 21696
2026-09-27T21:32:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1910 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:32:13 dist-test-slave-bxm0 krb5kdc[21794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:32:13 dist-test-slave-bxm0 krb5kdc[21794](info): set up 2 sockets
Sep 27 21:32:13 dist-test-slave-bxm0 krb5kdc[21794](info): commencing operation
krb5kdc: starting...
W20260927 21:32:14.994009 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:32:15 dist-test-slave-bxm0 krb5kdc[21794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:32:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:32:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:32:15.257687 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34931
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:32:15.584199 21807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:15.584507 21807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:15.584671 21807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:15.591887 21807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:32:15.592000 21807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:15.592087 21807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:15.592181 21807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:32:15.592263 21807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:32:15.602160 21807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34931
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:15.603503 21807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:15.604796 21807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:15.616966 21812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:15.616890 21813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:15.616946 21815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:15.617822 21807 server_base.cc:1034] running on GCE node
I20260927 21:32:15.618620 21807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:15.620056 21807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:15.621312 21807 hybrid_clock.cc:648] HybridClock initialized: now 1790544735621267 us; error 45 us; skew 500 ppm
I20260927 21:32:15.621819 21807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:15 dist-test-slave-bxm0 krb5kdc[21794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:15.630683 21807 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:32:15.632359 21807 webserver.cc:460] Webserver started at http://127.0.0.1:39061/ using document root <none> and password file <none>
I20260927 21:32:15.633198 21807 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:15.633351 21807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:15.633744 21807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:15.636859 21807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "1c43e4fe014d44ab9a73f77ab36faacc"
format_stamp: "Formatted at 2026-09-27 21:32:15 on dist-test-slave-bxm0"
server_key: "7235b1a0fc189bc387a605fc68c9d2c8"
server_key_iv: "34c7b998baca22357cc72160109a51e2"
server_key_version: "encryptionkey@0"
I20260927 21:32:15.637719 21807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "1c43e4fe014d44ab9a73f77ab36faacc"
format_stamp: "Formatted at 2026-09-27 21:32:15 on dist-test-slave-bxm0"
server_key: "7235b1a0fc189bc387a605fc68c9d2c8"
server_key_iv: "34c7b998baca22357cc72160109a51e2"
server_key_version: "encryptionkey@0"
I20260927 21:32:15.642992 21807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 21:32:15.646543 21822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:15.647935 21807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:32:15.648132 21807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "1c43e4fe014d44ab9a73f77ab36faacc"
format_stamp: "Formatted at 2026-09-27 21:32:15 on dist-test-slave-bxm0"
server_key: "7235b1a0fc189bc387a605fc68c9d2c8"
server_key_iv: "34c7b998baca22357cc72160109a51e2"
server_key_version: "encryptionkey@0"
I20260927 21:32:15.648344 21807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:15.659586 21807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:15.663792 21807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:15.664093 21807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:15.682899 21807 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34931
I20260927 21:32:15.682991 21873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34931 every 8 connection(s)
I20260927 21:32:15.684331 21807 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:32:15.688081 21874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:32:15.693328 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 21807
I20260927 21:32:15.693557 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:32:15.694023 388 external_mini_cluster.cc:1428] Setting key 581f9b8ad632b1e9ad8c2fd642e3f8e2
I20260927 21:32:15.699151 21874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c43e4fe014d44ab9a73f77ab36faacc: Bootstrap starting.
I20260927 21:32:15.703166 21874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c43e4fe014d44ab9a73f77ab36faacc: Neither blocks nor log segments found. Creating new log.
I20260927 21:32:15.704442 21874 log.cc:824] T 00000000000000000000000000000000 P 1c43e4fe014d44ab9a73f77ab36faacc: Log is configured to *not* fsync() on all Append() calls
I20260927 21:32:15.708069 21874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c43e4fe014d44ab9a73f77ab36faacc: No bootstrap required, opened a new log
Sep 27 21:32:15 dist-test-slave-bxm0 krb5kdc[21794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 21:32:15.716189 21874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c43e4fe014d44ab9a73f77ab36faacc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c43e4fe014d44ab9a73f77ab36faacc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34931 } }
I20260927 21:32:15.716605 21874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c43e4fe014d44ab9a73f77ab36faacc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:32:15.716729 21874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c43e4fe014d44ab9a73f77ab36faacc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c43e4fe014d44ab9a73f77ab36faacc, State: Initialized, Role: FOLLOWER
I20260927 21:32:15.717473 21874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c43e4fe014d44ab9a73f77ab36faacc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c43e4fe014d44ab9a73f77ab36faacc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34931 } }
I20260927 21:32:15.717844 21874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c43e4fe014d44ab9a73f77ab36faacc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:32:15.718046 21874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c43e4fe014d44ab9a73f77ab36faacc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:32:15.718241 21874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c43e4fe014d44ab9a73f77ab36faacc [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:32:15.718770 21877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:15.695906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:37487 (local address 127.0.0.1:34931)
0927 21:32:15.696807 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:15.696819 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:15.696867 (+ 48us) server_negotiation.cc:407] Connection header received
0927 21:32:15.698687 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:15.698712 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:15.699157 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:15.699555 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:15.701395 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:15.702783 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:15.704666 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:15.705761 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:15.705946 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:15.710702 (+ 4756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:15.710715 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:15.710732 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:15.710772 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:15.714110 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:15.715030 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:15.715037 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:15.715044 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:15.715229 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:15.715705 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:15.715718 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:15.715720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:15.716256 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:15.716651 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:15.717167 (+ 516us) server_negotiation.cc:299] Negotiation successful
0927 21:32:15.717501 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":422,"threads_started":1}
I20260927 21:32:15.720726 21874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c43e4fe014d44ab9a73f77ab36faacc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c43e4fe014d44ab9a73f77ab36faacc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34931 } }
I20260927 21:32:15.721272 21874 leader_election.cc:302] T 00000000000000000000000000000000 P 1c43e4fe014d44ab9a73f77ab36faacc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c43e4fe014d44ab9a73f77ab36faacc; no voters:
I20260927 21:32:15.722040 21874 leader_election.cc:288] T 00000000000000000000000000000000 P 1c43e4fe014d44ab9a73f77ab36faacc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:32:15.722139 21879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c43e4fe014d44ab9a73f77ab36faacc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:32:15.723047 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:32:15.724586 21874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c43e4fe014d44ab9a73f77ab36faacc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:32:15.729120 21879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c43e4fe014d44ab9a73f77ab36faacc [term 1 LEADER]: Becoming Leader. State: Replica: 1c43e4fe014d44ab9a73f77ab36faacc, State: Running, Role: LEADER
I20260927 21:32:15.729931 21879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c43e4fe014d44ab9a73f77ab36faacc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c43e4fe014d44ab9a73f77ab36faacc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34931 } }
I20260927 21:32:15.733721 21880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c43e4fe014d44ab9a73f77ab36faacc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c43e4fe014d44ab9a73f77ab36faacc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c43e4fe014d44ab9a73f77ab36faacc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34931 } } }
I20260927 21:32:15.734198 21880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c43e4fe014d44ab9a73f77ab36faacc [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:15.734828 21881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c43e4fe014d44ab9a73f77ab36faacc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c43e4fe014d44ab9a73f77ab36faacc. Latest consensus state: current_term: 1 leader_uuid: "1c43e4fe014d44ab9a73f77ab36faacc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c43e4fe014d44ab9a73f77ab36faacc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34931 } } }
I20260927 21:32:15.735157 21881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c43e4fe014d44ab9a73f77ab36faacc [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:15.757701 21894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:32:15.774031 21894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:32:15.783726 21877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:15.733140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:38665 (local address 127.0.0.1:34931)
0927 21:32:15.733331 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:15.733341 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:15.761333 (+ 27992us) server_negotiation.cc:407] Connection header received
0927 21:32:15.761583 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:15.761593 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:15.761755 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:15.761936 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:15.764716 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:15.766364 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:15.769013 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:15.770004 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:15.770136 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:15.774634 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:15.774641 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:15.774646 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:15.774694 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:15.779583 (+ 4889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:15.780719 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:15.780725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:15.780730 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:15.780890 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:15.781533 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:15.781539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:15.781542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:15.782079 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:15.782352 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:15.783112 (+ 760us) server_negotiation.cc:299] Negotiation successful
0927 21:32:15.783335 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:32:15.790994 21894 catalog_manager.cc:1347] Generated new cluster ID: 11473c1304bd49d0b25f841b2a5dcf04
I20260927 21:32:15.791115 21894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:32:15.816704 21894 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:32:15.818130 21894 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:32:15.828481 21894 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c43e4fe014d44ab9a73f77ab36faacc: Generated new TSK 0
I20260927 21:32:15.829576 21894 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:32:15.842406 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 21807
I20260927 21:32:15.843919 21869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:32:15.937105 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 21807
W20260927 21:32:15.954509 21792 connection.cc:535] client connection to 127.0.0.1:34931 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34931): Connection reset by peer (error 104)
2026-09-27T21:32:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2048 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:32:16 dist-test-slave-bxm0 krb5kdc[21907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:32:16 dist-test-slave-bxm0 krb5kdc[21907](info): set up 2 sockets
Sep 27 21:32:16 dist-test-slave-bxm0 krb5kdc[21907](info): commencing operation
krb5kdc: starting...
W20260927 21:32:17.050125 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.010s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:32:17 dist-test-slave-bxm0 krb5kdc[21907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:32:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:32:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:32:17.313180 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39121
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:32:17.646835 21920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:17.647121 21920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:17.647210 21920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:17.654511 21920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:32:17.654642 21920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:17.654690 21920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:17.654753 21920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:32:17.654796 21920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:32:17.664809 21920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39121
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:17.666091 21920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:17.667336 21920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:17.679191 21928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:17.679129 21925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:17.679198 21926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:17.680061 21920 server_base.cc:1034] running on GCE node
I20260927 21:32:17.680828 21920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:17.682262 21920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:17.683468 21920 hybrid_clock.cc:648] HybridClock initialized: now 1790544737683394 us; error 62 us; skew 500 ppm
I20260927 21:32:17.683772 21920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:17 dist-test-slave-bxm0 krb5kdc[21907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:17.686817 21920 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:32:17.688282 21920 webserver.cc:460] Webserver started at http://127.0.0.1:39343/ using document root <none> and password file <none>
I20260927 21:32:17.688967 21920 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:17.689113 21920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:17.689432 21920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:17.692479 21920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "f8beae323f22427ba0f3ad27ac2926b7"
format_stamp: "Formatted at 2026-09-27 21:32:17 on dist-test-slave-bxm0"
server_key: "c06a1824d7a4e24baa7e5e1af54f283f"
server_key_iv: "c5d411a95f6a96a938ce0af00f6f60af"
server_key_version: "encryptionkey@0"
I20260927 21:32:17.693271 21920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "f8beae323f22427ba0f3ad27ac2926b7"
format_stamp: "Formatted at 2026-09-27 21:32:17 on dist-test-slave-bxm0"
server_key: "c06a1824d7a4e24baa7e5e1af54f283f"
server_key_iv: "c5d411a95f6a96a938ce0af00f6f60af"
server_key_version: "encryptionkey@0"
I20260927 21:32:17.698642 21920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 21:32:17.702260 21935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:17.703604 21920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:32:17.703804 21920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "f8beae323f22427ba0f3ad27ac2926b7"
format_stamp: "Formatted at 2026-09-27 21:32:17 on dist-test-slave-bxm0"
server_key: "c06a1824d7a4e24baa7e5e1af54f283f"
server_key_iv: "c5d411a95f6a96a938ce0af00f6f60af"
server_key_version: "encryptionkey@0"
I20260927 21:32:17.704021 21920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:17.725170 21920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:17.729357 21920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:17.729705 21920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:17.748888 21920 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39121
I20260927 21:32:17.748881 21986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39121 every 8 connection(s)
I20260927 21:32:17.750434 21920 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:32:17.754192 21987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:32:17.758253 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 21920
I20260927 21:32:17.758427 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:32:17.758946 388 external_mini_cluster.cc:1428] Setting key ea40320efd8ec86180547430df650215
I20260927 21:32:17.767427 21987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8beae323f22427ba0f3ad27ac2926b7: Bootstrap starting.
I20260927 21:32:17.771131 21987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8beae323f22427ba0f3ad27ac2926b7: Neither blocks nor log segments found. Creating new log.
I20260927 21:32:17.772323 21987 log.cc:824] T 00000000000000000000000000000000 P f8beae323f22427ba0f3ad27ac2926b7: Log is configured to *not* fsync() on all Append() calls
I20260927 21:32:17.772894 21990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:17.760886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:58429 (local address 127.0.0.1:39121)
0927 21:32:17.761770 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:17.761782 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:17.761828 (+ 46us) server_negotiation.cc:407] Connection header received
0927 21:32:17.763502 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:17.763525 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:17.763966 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:17.764327 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:17.764913 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:17.766394 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:17.768238 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:17.768959 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:17.769104 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:17.770465 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:17.770472 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:17.770485 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:32:17.771001 (+ 516us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:17.771077 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:32:17.771106 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:17.771260 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:17.771574 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 21:32:17.771769 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":393,"threads_started":1}
I20260927 21:32:17.775486 21987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8beae323f22427ba0f3ad27ac2926b7: No bootstrap required, opened a new log
I20260927 21:32:17.776986 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:32:17.784548 21987 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8beae323f22427ba0f3ad27ac2926b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8beae323f22427ba0f3ad27ac2926b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260927 21:32:17.784946 21987 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8beae323f22427ba0f3ad27ac2926b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:32:17.785068 21987 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8beae323f22427ba0f3ad27ac2926b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8beae323f22427ba0f3ad27ac2926b7, State: Initialized, Role: FOLLOWER
I20260927 21:32:17.785805 21987 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8beae323f22427ba0f3ad27ac2926b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8beae323f22427ba0f3ad27ac2926b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260927 21:32:17.786106 21987 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8beae323f22427ba0f3ad27ac2926b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:32:17.786273 21987 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8beae323f22427ba0f3ad27ac2926b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:32:17.786459 21987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8beae323f22427ba0f3ad27ac2926b7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:32:17.788157 21990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:17.781387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:51317 (local address 127.0.0.1:39121)
0927 21:32:17.781581 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:17.781587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:17.781944 (+ 357us) server_negotiation.cc:407] Connection header received
0927 21:32:17.782093 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:17.782098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:17.782177 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:17.782297 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:17.782751 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:17.783897 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:17.785340 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:17.786027 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:17.786159 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:17.787377 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:17.787383 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:17.787388 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:32:17.787409 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:17.787480 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:32:17.787507 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:17.787618 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:17.787759 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 21:32:17.787816 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 21:32:17.789605 21987 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8beae323f22427ba0f3ad27ac2926b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8beae323f22427ba0f3ad27ac2926b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260927 21:32:17.790107 21987 leader_election.cc:302] T 00000000000000000000000000000000 P f8beae323f22427ba0f3ad27ac2926b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8beae323f22427ba0f3ad27ac2926b7; no voters:
I20260927 21:32:17.790642 21987 leader_election.cc:288] T 00000000000000000000000000000000 P f8beae323f22427ba0f3ad27ac2926b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:32:17.790771 21997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8beae323f22427ba0f3ad27ac2926b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:32:17.791469 21997 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8beae323f22427ba0f3ad27ac2926b7 [term 1 LEADER]: Becoming Leader. State: Replica: f8beae323f22427ba0f3ad27ac2926b7, State: Running, Role: LEADER
I20260927 21:32:17.791958 21997 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8beae323f22427ba0f3ad27ac2926b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8beae323f22427ba0f3ad27ac2926b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260927 21:32:17.792682 21987 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8beae323f22427ba0f3ad27ac2926b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:32:17.795007 21998 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8beae323f22427ba0f3ad27ac2926b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8beae323f22427ba0f3ad27ac2926b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8beae323f22427ba0f3ad27ac2926b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } } }
I20260927 21:32:17.795739 21998 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8beae323f22427ba0f3ad27ac2926b7 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:17.795737 21999 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8beae323f22427ba0f3ad27ac2926b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8beae323f22427ba0f3ad27ac2926b7. Latest consensus state: current_term: 1 leader_uuid: "f8beae323f22427ba0f3ad27ac2926b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8beae323f22427ba0f3ad27ac2926b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } } }
I20260927 21:32:17.796412 21999 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8beae323f22427ba0f3ad27ac2926b7 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:17.798051 22005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:32:17.805166 22005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:32:17.814291 22005 catalog_manager.cc:1347] Generated new cluster ID: edcf61e77c3d462cb2c7678880a697a5
I20260927 21:32:17.814414 22005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:32:17.835476 22005 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:32:17.836829 22005 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:32:17.849236 22005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8beae323f22427ba0f3ad27ac2926b7: Generated new TSK 0
I20260927 21:32:17.850368 22005 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:32:17.867661 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 21920
W20260927 21:32:17.868880 21982 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.24:58429
W20260927 21:32:17.869663 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 21920: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:32:17.869817 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 21920
W20260927 21:32:17.886639 21905 connection.cc:535] client connection to 127.0.0.1:39121 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39121): Connection reset by peer (error 104)
2026-09-27T21:32:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1932 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:32:17 dist-test-slave-bxm0 krb5kdc[22018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:32:17 dist-test-slave-bxm0 krb5kdc[22018](info): set up 2 sockets
Sep 27 21:32:17 dist-test-slave-bxm0 krb5kdc[22018](info): commencing operation
krb5kdc: starting...
W20260927 21:32:18.982237 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:32:19 dist-test-slave-bxm0 krb5kdc[22018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:32:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:32:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:32:19.244872 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40807
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:32:19.576330 22031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:19.576624 22031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:19.576797 22031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:19.584115 22031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:32:19.584236 22031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:19.584283 22031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:19.584381 22031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:32:19.584451 22031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:32:19.594372 22031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40807
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544390748963-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:19.595715 22031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:19.597046 22031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:19.608887 22036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:19.608961 22037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:19.609056 22039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:19.610028 22031 server_base.cc:1034] running on GCE node
I20260927 21:32:19.610767 22031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:19.612195 22031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:19.613462 22031 hybrid_clock.cc:648] HybridClock initialized: now 1790544739613413 us; error 50 us; skew 500 ppm
I20260927 21:32:19.613886 22031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:19 dist-test-slave-bxm0 krb5kdc[22018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:19.616941 22031 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:32:19.618489 22031 webserver.cc:460] Webserver started at http://127.0.0.1:43379/ using document root <none> and password file <none>
I20260927 21:32:19.619215 22031 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:19.619361 22031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:19.619699 22031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:19.623647 22031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "d3f94c286d0340d2abeea407878153e7"
format_stamp: "Formatted at 2026-09-27 21:32:19 on dist-test-slave-bxm0"
server_key: "f86edffe4925174ee755282b0dfb284e"
server_key_iv: "e74b00b1b605ae232a8f6baa7f8c7842"
server_key_version: "encryptionkey@0"
I20260927 21:32:19.624718 22031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "d3f94c286d0340d2abeea407878153e7"
format_stamp: "Formatted at 2026-09-27 21:32:19 on dist-test-slave-bxm0"
server_key: "f86edffe4925174ee755282b0dfb284e"
server_key_iv: "e74b00b1b605ae232a8f6baa7f8c7842"
server_key_version: "encryptionkey@0"
I20260927 21:32:19.632644 22031 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.010s sys 0.000s
I20260927 21:32:19.636135 22046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:19.637457 22031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 21:32:19.637683 22031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "d3f94c286d0340d2abeea407878153e7"
format_stamp: "Formatted at 2026-09-27 21:32:19 on dist-test-slave-bxm0"
server_key: "f86edffe4925174ee755282b0dfb284e"
server_key_iv: "e74b00b1b605ae232a8f6baa7f8c7842"
server_key_version: "encryptionkey@0"
I20260927 21:32:19.637898 22031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:19.643545 22031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:19.647624 22031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:19.647914 22031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:19.666568 22031 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40807
I20260927 21:32:19.666615 22097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40807 every 8 connection(s)
I20260927 21:32:19.668128 22031 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:32:19.670655 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 22031
I20260927 21:32:19.670825 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:32:19.671355 388 external_mini_cluster.cc:1428] Setting key d244f5d4630f3d64cd7f020127d10264
I20260927 21:32:19.672408 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:32:19.678920 22101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:19.673111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:40871 (local address 127.0.0.1:40807)
0927 21:32:19.673880 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:19.673891 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:19.673936 (+ 45us) server_negotiation.cc:407] Connection header received
0927 21:32:19.675460 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:19.675482 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:19.675821 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:19.676162 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:19.677221 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:19.677230 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:19.677243 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:32:19.677258 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:19.677331 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:32:19.677351 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:19.677523 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:19.677699 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 21:32:19.677936 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":334,"threads_started":1}
I20260927 21:32:19.682780 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:32:19.686241 22098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3f94c286d0340d2abeea407878153e7: Bootstrap starting.
I20260927 21:32:19.690477 22098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3f94c286d0340d2abeea407878153e7: Neither blocks nor log segments found. Creating new log.
I20260927 21:32:19.690759 22101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:19.688134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:33645 (local address 127.0.0.1:40807)
0927 21:32:19.688292 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:19.688297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:19.688648 (+ 351us) server_negotiation.cc:407] Connection header received
0927 21:32:19.688819 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:19.688822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:19.688873 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:19.688952 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:19.689906 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:19.689910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:19.689913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:32:19.689927 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:19.689987 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:32:19.690002 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:19.690088 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:19.690277 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 21:32:19.690332 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:32:19.691746 22098 log.cc:824] T 00000000000000000000000000000000 P d3f94c286d0340d2abeea407878153e7: Log is configured to *not* fsync() on all Append() calls
I20260927 21:32:19.694569 22098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3f94c286d0340d2abeea407878153e7: No bootstrap required, opened a new log
I20260927 21:32:19.702306 22098 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3f94c286d0340d2abeea407878153e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f94c286d0340d2abeea407878153e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40807 } }
I20260927 21:32:19.702742 22098 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3f94c286d0340d2abeea407878153e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:32:19.702867 22098 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3f94c286d0340d2abeea407878153e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3f94c286d0340d2abeea407878153e7, State: Initialized, Role: FOLLOWER
I20260927 21:32:19.703466 22098 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3f94c286d0340d2abeea407878153e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f94c286d0340d2abeea407878153e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40807 } }
I20260927 21:32:19.703724 22098 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3f94c286d0340d2abeea407878153e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:32:19.703882 22098 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3f94c286d0340d2abeea407878153e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:32:19.704053 22098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3f94c286d0340d2abeea407878153e7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:32:19.706426 22098 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3f94c286d0340d2abeea407878153e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f94c286d0340d2abeea407878153e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40807 } }
I20260927 21:32:19.706956 22098 leader_election.cc:302] T 00000000000000000000000000000000 P d3f94c286d0340d2abeea407878153e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3f94c286d0340d2abeea407878153e7; no voters:
I20260927 21:32:19.707479 22098 leader_election.cc:288] T 00000000000000000000000000000000 P d3f94c286d0340d2abeea407878153e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:32:19.707623 22108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3f94c286d0340d2abeea407878153e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:32:19.708323 22108 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3f94c286d0340d2abeea407878153e7 [term 1 LEADER]: Becoming Leader. State: Replica: d3f94c286d0340d2abeea407878153e7, State: Running, Role: LEADER
I20260927 21:32:19.708873 22108 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3f94c286d0340d2abeea407878153e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f94c286d0340d2abeea407878153e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40807 } }
I20260927 21:32:19.709676 22098 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3f94c286d0340d2abeea407878153e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:32:19.711648 22109 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3f94c286d0340d2abeea407878153e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3f94c286d0340d2abeea407878153e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f94c286d0340d2abeea407878153e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40807 } } }
I20260927 21:32:19.711826 22110 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3f94c286d0340d2abeea407878153e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3f94c286d0340d2abeea407878153e7. Latest consensus state: current_term: 1 leader_uuid: "d3f94c286d0340d2abeea407878153e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f94c286d0340d2abeea407878153e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40807 } } }
I20260927 21:32:19.712183 22109 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3f94c286d0340d2abeea407878153e7 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:19.712180 22110 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3f94c286d0340d2abeea407878153e7 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:19.715166 22116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:32:19.720388 22116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:32:19.729934 22116 catalog_manager.cc:1347] Generated new cluster ID: 8ad64cfefb7b454ab691c325a2be4b21
I20260927 21:32:19.730077 22116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:32:19.745281 22116 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:32:19.745831 22116 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:32:19.755359 22116 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3f94c286d0340d2abeea407878153e7: Generated new TSK 0
I20260927 21:32:19.756281 22116 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:32:19.778576 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 22031
W20260927 21:32:19.780135 22093 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.24:40871
W20260927 21:32:19.780968 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 22031: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:32:19.781142 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 22031
2026-09-27T21:32:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1912 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:32:19 dist-test-slave-bxm0 krb5kdc[22129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:32:19 dist-test-slave-bxm0 krb5kdc[22129](info): set up 2 sockets
Sep 27 21:32:19 dist-test-slave-bxm0 krb5kdc[22129](info): commencing operation
krb5kdc: starting...
W20260927 21:32:20.902042 388 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.005s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:32:21 dist-test-slave-bxm0 krb5kdc[22129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:32:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:32:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:32:21.161294 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36343
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:32:21.492149 22142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:21.492537 22142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:21.492707 22142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:21.500070 22142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:32:21.500243 22142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:21.500351 22142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:21.500458 22142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:32:21.500532 22142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:32:21.510723 22142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36343
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:21.512107 22142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:21.513404 22142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:21.525241 22147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:21.525336 22148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:21.525251 22150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:21.526293 22142 server_base.cc:1034] running on GCE node
I20260927 21:32:21.527469 22142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:21.528885 22142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:21.530118 22142 hybrid_clock.cc:648] HybridClock initialized: now 1790544741530096 us; error 55 us; skew 500 ppm
I20260927 21:32:21.530467 22142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:21 dist-test-slave-bxm0 krb5kdc[22129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:21.540608 22142 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:32:21.542351 22142 webserver.cc:460] Webserver started at http://127.0.97.62:32845/ using document root <none> and password file <none>
I20260927 21:32:21.543085 22142 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:21.543243 22142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:21.543560 22142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:21.546657 22142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "e76064b47abd4f9dbaedec51d12f3376"
format_stamp: "Formatted at 2026-09-27 21:32:21 on dist-test-slave-bxm0"
server_key: "02ffd5ece0ce2f209181c0d23970e104"
server_key_iv: "735e26fda3dbdb44429a57c660feee08"
server_key_version: "encryptionkey@0"
I20260927 21:32:21.547415 22142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "e76064b47abd4f9dbaedec51d12f3376"
format_stamp: "Formatted at 2026-09-27 21:32:21 on dist-test-slave-bxm0"
server_key: "02ffd5ece0ce2f209181c0d23970e104"
server_key_iv: "735e26fda3dbdb44429a57c660feee08"
server_key_version: "encryptionkey@0"
I20260927 21:32:21.552675 22142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:32:21.556100 22157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:21.557339 22142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:32:21.557531 22142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "e76064b47abd4f9dbaedec51d12f3376"
format_stamp: "Formatted at 2026-09-27 21:32:21 on dist-test-slave-bxm0"
server_key: "02ffd5ece0ce2f209181c0d23970e104"
server_key_iv: "735e26fda3dbdb44429a57c660feee08"
server_key_version: "encryptionkey@0"
I20260927 21:32:21.557760 22142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:21.570659 22142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:21.574409 22142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:21.574712 22142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:21.593331 22142 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36343
I20260927 21:32:21.593350 22208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36343 every 8 connection(s)
I20260927 21:32:21.594816 22142 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:32:21.596453 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 22142
I20260927 21:32:21.596647 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:32:21.597069 388 external_mini_cluster.cc:1428] Setting key 28d5ffc6cae4050abbabeaf8135acb2e
I20260927 21:32:21.599150 22209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 21:32:21 dist-test-slave-bxm0 krb5kdc[22129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:32:21.612382 22209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e76064b47abd4f9dbaedec51d12f3376: Bootstrap starting.
I20260927 21:32:21.616712 22209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e76064b47abd4f9dbaedec51d12f3376: Neither blocks nor log segments found. Creating new log.
I20260927 21:32:21.618050 22209 log.cc:824] T 00000000000000000000000000000000 P e76064b47abd4f9dbaedec51d12f3376: Log is configured to *not* fsync() on all Append() calls
I20260927 21:32:21.621088 22209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e76064b47abd4f9dbaedec51d12f3376: No bootstrap required, opened a new log
I20260927 21:32:21.621685 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:21.598826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55579 (local address 127.0.97.62:36343)
0927 21:32:21.599610 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:21.599626 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:21.599674 (+ 48us) server_negotiation.cc:407] Connection header received
0927 21:32:21.602004 (+ 2330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:21.602030 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:21.602553 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:21.602997 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:21.604265 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:21.606353 (+ 2088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:21.607640 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:21.608687 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:21.608890 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:21.612946 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:21.612956 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:21.612976 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:21.613023 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:21.617032 (+ 4009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:21.617936 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:21.617944 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:21.617953 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:21.618176 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:21.618626 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:21.618634 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:21.618640 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:21.619193 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:21.619519 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:21.620034 (+ 515us) server_negotiation.cc:299] Negotiation successful
0927 21:32:21.620367 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":343,"threads_started":1}
I20260927 21:32:21.626734 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:32:21.631541 22209 raft_consensus.cc:348] T 00000000000000000000000000000000 P e76064b47abd4f9dbaedec51d12f3376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76064b47abd4f9dbaedec51d12f3376" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36343 } }
I20260927 21:32:21.632082 22209 raft_consensus.cc:374] T 00000000000000000000000000000000 P e76064b47abd4f9dbaedec51d12f3376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:32:21.632179 22209 raft_consensus.cc:729] T 00000000000000000000000000000000 P e76064b47abd4f9dbaedec51d12f3376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e76064b47abd4f9dbaedec51d12f3376, State: Initialized, Role: FOLLOWER
I20260927 21:32:21.632891 22209 consensus_queue.cc:260] T 00000000000000000000000000000000 P e76064b47abd4f9dbaedec51d12f3376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76064b47abd4f9dbaedec51d12f3376" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36343 } }
I20260927 21:32:21.633111 22209 raft_consensus.cc:388] T 00000000000000000000000000000000 P e76064b47abd4f9dbaedec51d12f3376 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:32:21.633265 22209 raft_consensus.cc:482] T 00000000000000000000000000000000 P e76064b47abd4f9dbaedec51d12f3376 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:32:21.633436 22209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e76064b47abd4f9dbaedec51d12f3376 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:32:21.636336 22209 raft_consensus.cc:504] T 00000000000000000000000000000000 P e76064b47abd4f9dbaedec51d12f3376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76064b47abd4f9dbaedec51d12f3376" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36343 } }
I20260927 21:32:21.636930 22209 leader_election.cc:302] T 00000000000000000000000000000000 P e76064b47abd4f9dbaedec51d12f3376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e76064b47abd4f9dbaedec51d12f3376; no voters:
I20260927 21:32:21.637696 22209 leader_election.cc:288] T 00000000000000000000000000000000 P e76064b47abd4f9dbaedec51d12f3376 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:32:21.637831 22219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e76064b47abd4f9dbaedec51d12f3376 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:32:21.639667 22209 sys_catalog.cc:564] T 00000000000000000000000000000000 P e76064b47abd4f9dbaedec51d12f3376 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:32:21.642702 22219 raft_consensus.cc:686] T 00000000000000000000000000000000 P e76064b47abd4f9dbaedec51d12f3376 [term 1 LEADER]: Becoming Leader. State: Replica: e76064b47abd4f9dbaedec51d12f3376, State: Running, Role: LEADER
I20260927 21:32:21.643385 22219 consensus_queue.cc:237] T 00000000000000000000000000000000 P e76064b47abd4f9dbaedec51d12f3376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76064b47abd4f9dbaedec51d12f3376" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36343 } }
I20260927 21:32:21.646670 22220 sys_catalog.cc:455] T 00000000000000000000000000000000 P e76064b47abd4f9dbaedec51d12f3376 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e76064b47abd4f9dbaedec51d12f3376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76064b47abd4f9dbaedec51d12f3376" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36343 } } }
I20260927 21:32:21.647149 22220 sys_catalog.cc:458] T 00000000000000000000000000000000 P e76064b47abd4f9dbaedec51d12f3376 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:21.647734 22221 sys_catalog.cc:455] T 00000000000000000000000000000000 P e76064b47abd4f9dbaedec51d12f3376 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e76064b47abd4f9dbaedec51d12f3376. Latest consensus state: current_term: 1 leader_uuid: "e76064b47abd4f9dbaedec51d12f3376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e76064b47abd4f9dbaedec51d12f3376" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36343 } } }
I20260927 21:32:21.648149 22221 sys_catalog.cc:458] T 00000000000000000000000000000000 P e76064b47abd4f9dbaedec51d12f3376 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:21.650391 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:21.631540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46497 (local address 127.0.97.62:36343)
0927 21:32:21.631712 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:21.631717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:21.632251 (+ 534us) server_negotiation.cc:407] Connection header received
0927 21:32:21.632447 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:21.632454 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:21.632545 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:21.632650 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:21.634237 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:21.635440 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:21.636732 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:21.637374 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:21.637553 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:21.641261 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:21.641271 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:21.641282 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:21.641342 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:21.645474 (+ 4132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:21.646606 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:21.646612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:21.646615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:21.646880 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:21.647491 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:21.647496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:21.647499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:21.647893 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:21.648233 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:21.649642 (+ 1409us) server_negotiation.cc:299] Negotiation successful
0927 21:32:21.649882 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 21:32:21.658833 22228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:32:21.664997 22228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:32:21.674369 22228 catalog_manager.cc:1347] Generated new cluster ID: 66ac70eaf7f64e3380f2904032ded8c5
I20260927 21:32:21.674504 22228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:32:21.695910 22228 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:32:21.697948 22228 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:32:21.710129 22228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e76064b47abd4f9dbaedec51d12f3376: Generated new TSK 0
I20260927 21:32:21.711097 22228 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:32:21.729380 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 22142
I20260927 21:32:21.730582 22204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:32:21.823968 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 22142
W20260927 21:32:21.841295 22127 connection.cc:535] client connection to 127.0.97.62:36343 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:36343): Connection reset by peer (error 104)
2026-09-27T21:32:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2041 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:32:21 dist-test-slave-bxm0 krb5kdc[22241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:32:21 dist-test-slave-bxm0 krb5kdc[22241](info): set up 2 sockets
Sep 27 21:32:21 dist-test-slave-bxm0 krb5kdc[22241](info): commencing operation
krb5kdc: starting...
W20260927 21:32:22.934084 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:32:23 dist-test-slave-bxm0 krb5kdc[22241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:32:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:32:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:32:23.192370 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43965
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:32:23.527325 22254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:23.527694 22254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:23.527870 22254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:23.535302 22254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:32:23.535471 22254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:23.535578 22254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:23.535701 22254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:32:23.535780 22254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:32:23.545881 22254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43965
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:23.547241 22254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:23.548550 22254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:23.560369 22259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:23.560439 22260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:23.560509 22262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:23.561558 22254 server_base.cc:1034] running on GCE node
I20260927 21:32:23.562438 22254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:23.563876 22254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:23.565079 22254 hybrid_clock.cc:648] HybridClock initialized: now 1790544743565037 us; error 31 us; skew 500 ppm
I20260927 21:32:23.565431 22254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:23 dist-test-slave-bxm0 krb5kdc[22241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:23.568663 22254 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:32:23.570284 22254 webserver.cc:460] Webserver started at http://127.0.97.62:43633/ using document root <none> and password file <none>
I20260927 21:32:23.570977 22254 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:23.571131 22254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:23.571444 22254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:23.574496 22254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "7c231235206e40ceaad17f0e0c71f4de"
format_stamp: "Formatted at 2026-09-27 21:32:23 on dist-test-slave-bxm0"
server_key: "58970fcf5fb1b90861cb47d7538a62b8"
server_key_iv: "c02359feae96076571efa99400a9a37e"
server_key_version: "encryptionkey@0"
I20260927 21:32:23.575289 22254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "7c231235206e40ceaad17f0e0c71f4de"
format_stamp: "Formatted at 2026-09-27 21:32:23 on dist-test-slave-bxm0"
server_key: "58970fcf5fb1b90861cb47d7538a62b8"
server_key_iv: "c02359feae96076571efa99400a9a37e"
server_key_version: "encryptionkey@0"
I20260927 21:32:23.580418 22254 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:32:23.583915 22269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:23.585275 22254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 21:32:23.585464 22254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "7c231235206e40ceaad17f0e0c71f4de"
format_stamp: "Formatted at 2026-09-27 21:32:23 on dist-test-slave-bxm0"
server_key: "58970fcf5fb1b90861cb47d7538a62b8"
server_key_iv: "c02359feae96076571efa99400a9a37e"
server_key_version: "encryptionkey@0"
I20260927 21:32:23.585721 22254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:23.607852 22254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:23.612450 22254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:23.612746 22254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:23.642210 22254 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43965
I20260927 21:32:23.642249 22320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43965 every 8 connection(s)
I20260927 21:32:23.643672 22254 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:32:23.647469 22321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:32:23.648090 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 22254
I20260927 21:32:23.648262 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:32:23.648710 388 external_mini_cluster.cc:1428] Setting key 72bd25e5759b93224be16dfd79a04892
I20260927 21:32:23.661020 22321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c231235206e40ceaad17f0e0c71f4de: Bootstrap starting.
I20260927 21:32:23.662628 22324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:23.650441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49721 (local address 127.0.97.62:43965)
0927 21:32:23.651235 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:23.651248 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:23.651287 (+ 39us) server_negotiation.cc:407] Connection header received
0927 21:32:23.653198 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:23.653252 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:23.653752 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:23.654107 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:23.654731 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:23.656004 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:23.657983 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:23.658689 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:23.658875 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:23.660253 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:23.660267 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:23.660281 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:32:23.660843 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:23.660940 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:32:23.660966 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:23.661105 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:23.661434 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 21:32:23.661651 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":373,"threads_started":1}
I20260927 21:32:23.664369 22321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c231235206e40ceaad17f0e0c71f4de: Neither blocks nor log segments found. Creating new log.
I20260927 21:32:23.665532 22321 log.cc:824] T 00000000000000000000000000000000 P 7c231235206e40ceaad17f0e0c71f4de: Log is configured to *not* fsync() on all Append() calls
I20260927 21:32:23.666486 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:32:23.668869 22321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c231235206e40ceaad17f0e0c71f4de: No bootstrap required, opened a new log
I20260927 21:32:23.678350 22324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:23.671658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51623 (local address 127.0.97.62:43965)
0927 21:32:23.671848 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:23.671854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:23.672303 (+ 449us) server_negotiation.cc:407] Connection header received
0927 21:32:23.672436 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:23.672442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:23.672518 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:23.672627 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:23.673083 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:23.674252 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:23.675570 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:23.676276 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:23.676413 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:23.677556 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:23.677562 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:23.677566 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:32:23.677589 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:23.677657 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:32:23.677684 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:23.677802 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:23.677966 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 21:32:23.678029 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 21:32:23.678877 22321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c231235206e40ceaad17f0e0c71f4de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c231235206e40ceaad17f0e0c71f4de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43965 } }
I20260927 21:32:23.679257 22321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c231235206e40ceaad17f0e0c71f4de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:32:23.679388 22321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c231235206e40ceaad17f0e0c71f4de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c231235206e40ceaad17f0e0c71f4de, State: Initialized, Role: FOLLOWER
I20260927 21:32:23.680029 22321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c231235206e40ceaad17f0e0c71f4de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c231235206e40ceaad17f0e0c71f4de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43965 } }
I20260927 21:32:23.680281 22321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c231235206e40ceaad17f0e0c71f4de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:32:23.680444 22321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c231235206e40ceaad17f0e0c71f4de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:32:23.680634 22321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c231235206e40ceaad17f0e0c71f4de [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:32:23.682885 22321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c231235206e40ceaad17f0e0c71f4de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c231235206e40ceaad17f0e0c71f4de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43965 } }
I20260927 21:32:23.683398 22321 leader_election.cc:302] T 00000000000000000000000000000000 P 7c231235206e40ceaad17f0e0c71f4de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c231235206e40ceaad17f0e0c71f4de; no voters:
I20260927 21:32:23.683980 22321 leader_election.cc:288] T 00000000000000000000000000000000 P 7c231235206e40ceaad17f0e0c71f4de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:32:23.684125 22331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c231235206e40ceaad17f0e0c71f4de [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:32:23.684813 22331 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c231235206e40ceaad17f0e0c71f4de [term 1 LEADER]: Becoming Leader. State: Replica: 7c231235206e40ceaad17f0e0c71f4de, State: Running, Role: LEADER
I20260927 21:32:23.685389 22331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c231235206e40ceaad17f0e0c71f4de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c231235206e40ceaad17f0e0c71f4de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43965 } }
I20260927 21:32:23.686153 22321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c231235206e40ceaad17f0e0c71f4de [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:32:23.688750 22333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c231235206e40ceaad17f0e0c71f4de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c231235206e40ceaad17f0e0c71f4de. Latest consensus state: current_term: 1 leader_uuid: "7c231235206e40ceaad17f0e0c71f4de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c231235206e40ceaad17f0e0c71f4de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43965 } } }
I20260927 21:32:23.689302 22333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c231235206e40ceaad17f0e0c71f4de [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:23.689162 22332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c231235206e40ceaad17f0e0c71f4de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c231235206e40ceaad17f0e0c71f4de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c231235206e40ceaad17f0e0c71f4de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43965 } } }
I20260927 21:32:23.689489 22332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c231235206e40ceaad17f0e0c71f4de [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:23.690856 22340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:32:23.696410 22340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:32:23.705274 22340 catalog_manager.cc:1347] Generated new cluster ID: 427cf124d1244e11acfeca1fd12e6356
I20260927 21:32:23.705418 22340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:32:23.725392 22340 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:32:23.727201 22340 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:32:23.744807 22340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c231235206e40ceaad17f0e0c71f4de: Generated new TSK 0
I20260927 21:32:23.745726 22340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:32:23.754463 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 22254
W20260927 21:32:23.755715 22316 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49721
W20260927 21:32:23.756496 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 22254: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:32:23.756656 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 22254
W20260927 21:32:23.773798 22239 connection.cc:535] client connection to 127.0.97.62:43965 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:43965): Connection reset by peer (error 104)
2026-09-27T21:32:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1932 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:32:23 dist-test-slave-bxm0 krb5kdc[22352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:32:23 dist-test-slave-bxm0 krb5kdc[22352](info): set up 2 sockets
Sep 27 21:32:23 dist-test-slave-bxm0 krb5kdc[22352](info): commencing operation
krb5kdc: starting...
W20260927 21:32:24.854117 388 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:32:25 dist-test-slave-bxm0 krb5kdc[22352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:32:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:32:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:32:25.108866 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45367
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:32:25.442605 22365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:25.442898 22365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:25.443033 22365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:25.450358 22365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:32:25.450471 22365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:25.450599 22365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:25.450699 22365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:32:25.450778 22365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:32:25.460944 22365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45367
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:25.462394 22365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:25.463732 22365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:25.475718 22371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:25.475661 22373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:25.475760 22370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:25.476377 22365 server_base.cc:1034] running on GCE node
I20260927 21:32:25.477226 22365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:25.478798 22365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:25.480007 22365 hybrid_clock.cc:648] HybridClock initialized: now 1790544745479967 us; error 40 us; skew 500 ppm
I20260927 21:32:25.480368 22365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:25 dist-test-slave-bxm0 krb5kdc[22352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:25.483727 22365 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:32:25.485266 22365 webserver.cc:460] Webserver started at http://127.0.97.62:37571/ using document root <none> and password file <none>
I20260927 21:32:25.486018 22365 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:25.486145 22365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:25.486454 22365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:25.489410 22365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "4a8af282255d4ed096ac0695d98c277a"
format_stamp: "Formatted at 2026-09-27 21:32:25 on dist-test-slave-bxm0"
server_key: "e170dfadd70ffef11c7186a44b7ab5ee"
server_key_iv: "08e090cb99c7e3495bde24c18410db2d"
server_key_version: "encryptionkey@0"
I20260927 21:32:25.490248 22365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "4a8af282255d4ed096ac0695d98c277a"
format_stamp: "Formatted at 2026-09-27 21:32:25 on dist-test-slave-bxm0"
server_key: "e170dfadd70ffef11c7186a44b7ab5ee"
server_key_iv: "08e090cb99c7e3495bde24c18410db2d"
server_key_version: "encryptionkey@0"
I20260927 21:32:25.495311 22365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 21:32:25.498704 22380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:25.500002 22365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 21:32:25.500180 22365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "4a8af282255d4ed096ac0695d98c277a"
format_stamp: "Formatted at 2026-09-27 21:32:25 on dist-test-slave-bxm0"
server_key: "e170dfadd70ffef11c7186a44b7ab5ee"
server_key_iv: "08e090cb99c7e3495bde24c18410db2d"
server_key_version: "encryptionkey@0"
I20260927 21:32:25.500367 22365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:25.506134 22365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:25.510449 22365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:25.510751 22365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:25.529024 22365 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45367
I20260927 21:32:25.529069 22431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45367 every 8 connection(s)
I20260927 21:32:25.530479 22365 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:32:25.534087 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 22365
I20260927 21:32:25.534260 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:32:25.534401 22432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:32:25.534791 388 external_mini_cluster.cc:1428] Setting key cb5af587fd25d4db365bac8e61509fc4
I20260927 21:32:25.542471 22435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:25.536526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60037 (local address 127.0.97.62:45367)
0927 21:32:25.537328 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:25.537339 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:25.537380 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:32:25.539195 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:25.539217 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:25.539540 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:25.539848 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:25.540937 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:25.540944 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:25.540956 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:32:25.540969 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:25.541026 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:32:25.541048 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:25.541213 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:25.541330 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 21:32:25.541526 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":356,"threads_started":1}
I20260927 21:32:25.545991 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:32:25.548173 22432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a8af282255d4ed096ac0695d98c277a: Bootstrap starting.
I20260927 21:32:25.552232 22432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a8af282255d4ed096ac0695d98c277a: Neither blocks nor log segments found. Creating new log.
I20260927 21:32:25.553588 22435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:25.551015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51297 (local address 127.0.97.62:45367)
0927 21:32:25.551205 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:25.551211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:25.551679 (+ 468us) server_negotiation.cc:407] Connection header received
0927 21:32:25.551813 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:25.551818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:25.551881 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:25.551996 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:25.552933 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:25.552939 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:25.552943 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:32:25.552961 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:25.553007 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:32:25.553027 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:25.553153 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:25.553216 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 21:32:25.553273 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:32:25.553580 22432 log.cc:824] T 00000000000000000000000000000000 P 4a8af282255d4ed096ac0695d98c277a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:32:25.556411 22432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a8af282255d4ed096ac0695d98c277a: No bootstrap required, opened a new log
I20260927 21:32:25.563524 22432 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a8af282255d4ed096ac0695d98c277a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8af282255d4ed096ac0695d98c277a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45367 } }
I20260927 21:32:25.563922 22432 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a8af282255d4ed096ac0695d98c277a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:32:25.564041 22432 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a8af282255d4ed096ac0695d98c277a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a8af282255d4ed096ac0695d98c277a, State: Initialized, Role: FOLLOWER
I20260927 21:32:25.564661 22432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a8af282255d4ed096ac0695d98c277a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8af282255d4ed096ac0695d98c277a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45367 } }
I20260927 21:32:25.564867 22432 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a8af282255d4ed096ac0695d98c277a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:32:25.564965 22432 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a8af282255d4ed096ac0695d98c277a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:32:25.565083 22432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a8af282255d4ed096ac0695d98c277a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:32:25.567250 22432 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a8af282255d4ed096ac0695d98c277a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8af282255d4ed096ac0695d98c277a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45367 } }
I20260927 21:32:25.567685 22432 leader_election.cc:302] T 00000000000000000000000000000000 P 4a8af282255d4ed096ac0695d98c277a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a8af282255d4ed096ac0695d98c277a; no voters:
I20260927 21:32:25.568238 22432 leader_election.cc:288] T 00000000000000000000000000000000 P 4a8af282255d4ed096ac0695d98c277a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:32:25.568391 22442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a8af282255d4ed096ac0695d98c277a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:32:25.569064 22442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a8af282255d4ed096ac0695d98c277a [term 1 LEADER]: Becoming Leader. State: Replica: 4a8af282255d4ed096ac0695d98c277a, State: Running, Role: LEADER
I20260927 21:32:25.569675 22442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a8af282255d4ed096ac0695d98c277a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8af282255d4ed096ac0695d98c277a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45367 } }
I20260927 21:32:25.570431 22432 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a8af282255d4ed096ac0695d98c277a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:32:25.573094 22443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a8af282255d4ed096ac0695d98c277a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a8af282255d4ed096ac0695d98c277a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8af282255d4ed096ac0695d98c277a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45367 } } }
I20260927 21:32:25.573068 22444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a8af282255d4ed096ac0695d98c277a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a8af282255d4ed096ac0695d98c277a. Latest consensus state: current_term: 1 leader_uuid: "4a8af282255d4ed096ac0695d98c277a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8af282255d4ed096ac0695d98c277a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45367 } } }
I20260927 21:32:25.573765 22443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a8af282255d4ed096ac0695d98c277a [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:25.573809 22444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a8af282255d4ed096ac0695d98c277a [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:25.581393 22449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:32:25.585794 22449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:32:25.594542 22449 catalog_manager.cc:1347] Generated new cluster ID: 6cfdd32ea2a24261a9a9833810553560
I20260927 21:32:25.594669 22449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:32:25.613801 22449 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:32:25.614256 22449 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:32:25.629253 22449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a8af282255d4ed096ac0695d98c277a: Generated new TSK 0
I20260927 21:32:25.630409 22449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:32:25.646646 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 22365
W20260927 21:32:25.647836 22427 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60037
W20260927 21:32:25.648545 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 22365: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:32:25.648695 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 22365
2026-09-27T21:32:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1891 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:32:25 dist-test-slave-bxm0 krb5kdc[22463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:32:25 dist-test-slave-bxm0 krb5kdc[22463](info): set up 2 sockets
Sep 27 21:32:25 dist-test-slave-bxm0 krb5kdc[22463](info): commencing operation
krb5kdc: starting...
W20260927 21:32:26.742192 388 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:32:26 dist-test-slave-bxm0 krb5kdc[22463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:32:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:32:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:32:27.004483 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33429
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:32:27.335909 22476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:27.336216 22476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:27.336390 22476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:27.343643 22476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:32:27.343751 22476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:27.343878 22476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:27.344013 22476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:32:27.344072 22476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:32:27.353981 22476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:27.355319 22476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:27.356618 22476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:27.368310 22484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:27.368276 22482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:27.368371 22481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:27.369232 22476 server_base.cc:1034] running on GCE node
I20260927 21:32:27.370106 22476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:27.371502 22476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:27.372731 22476 hybrid_clock.cc:648] HybridClock initialized: now 1790544747372674 us; error 53 us; skew 500 ppm
I20260927 21:32:27.373100 22476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:27 dist-test-slave-bxm0 krb5kdc[22463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:27.382934 22476 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:32:27.384485 22476 webserver.cc:460] Webserver started at http://127.0.97.62:36549/ using document root <none> and password file <none>
I20260927 21:32:27.385250 22476 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:27.385387 22476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:27.385792 22476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:27.388841 22476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "01d34370795342edb45c315a226b8cfd"
format_stamp: "Formatted at 2026-09-27 21:32:27 on dist-test-slave-bxm0"
server_key: "eb9ef02b57841a6826b3785bb6df8fb5"
server_key_iv: "c884270799551a4046f7da6c5eaf8724"
server_key_version: "encryptionkey@0"
I20260927 21:32:27.389699 22476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "01d34370795342edb45c315a226b8cfd"
format_stamp: "Formatted at 2026-09-27 21:32:27 on dist-test-slave-bxm0"
server_key: "eb9ef02b57841a6826b3785bb6df8fb5"
server_key_iv: "c884270799551a4046f7da6c5eaf8724"
server_key_version: "encryptionkey@0"
I20260927 21:32:27.394779 22476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260927 21:32:27.398176 22491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:27.399622 22476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:32:27.399817 22476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "01d34370795342edb45c315a226b8cfd"
format_stamp: "Formatted at 2026-09-27 21:32:27 on dist-test-slave-bxm0"
server_key: "eb9ef02b57841a6826b3785bb6df8fb5"
server_key_iv: "c884270799551a4046f7da6c5eaf8724"
server_key_version: "encryptionkey@0"
I20260927 21:32:27.400019 22476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:27.418429 22476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:27.423074 22476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:27.423372 22476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:27.442277 22476 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33429
I20260927 21:32:27.442309 22542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33429 every 8 connection(s)
I20260927 21:32:27.443692 22476 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:32:27.447383 22543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:32:27.450104 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 22476
I20260927 21:32:27.450310 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:32:27.450726 388 external_mini_cluster.cc:1428] Setting key c1b4da017dae30420c9952719cf5a59f
I20260927 21:32:27.460348 22543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01d34370795342edb45c315a226b8cfd: Bootstrap starting.
I20260927 21:32:27.464160 22543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01d34370795342edb45c315a226b8cfd: Neither blocks nor log segments found. Creating new log.
I20260927 21:32:27.465212 22543 log.cc:824] T 00000000000000000000000000000000 P 01d34370795342edb45c315a226b8cfd: Log is configured to *not* fsync() on all Append() calls
Sep 27 21:32:27 dist-test-slave-bxm0 krb5kdc[22463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:32:27.468690 22543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01d34370795342edb45c315a226b8cfd: No bootstrap required, opened a new log
I20260927 21:32:27.476511 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:27.452585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56819 (local address 127.0.97.62:33429)
0927 21:32:27.453410 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:27.453426 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:27.453475 (+ 49us) server_negotiation.cc:407] Connection header received
0927 21:32:27.455902 (+ 2427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:27.455931 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:27.456467 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:27.456928 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:27.458322 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:27.460276 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:27.461685 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:27.462799 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:27.462982 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:27.467358 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:27.467370 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:27.467393 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:27.467444 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:27.471653 (+ 4209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:27.472417 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:27.472426 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:27.472435 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:27.472677 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:27.473065 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:27.473070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:27.473074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:27.473842 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:27.474126 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:27.474528 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 21:32:27.475060 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":352,"threads_started":1}
I20260927 21:32:27.479591 22543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01d34370795342edb45c315a226b8cfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01d34370795342edb45c315a226b8cfd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33429 } }
I20260927 21:32:27.480095 22543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01d34370795342edb45c315a226b8cfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:32:27.480203 22543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01d34370795342edb45c315a226b8cfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d34370795342edb45c315a226b8cfd, State: Initialized, Role: FOLLOWER
I20260927 21:32:27.480744 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:32:27.480955 22543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01d34370795342edb45c315a226b8cfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01d34370795342edb45c315a226b8cfd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33429 } }
I20260927 21:32:27.481237 22543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01d34370795342edb45c315a226b8cfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:32:27.481364 22543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01d34370795342edb45c315a226b8cfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:32:27.481518 22543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01d34370795342edb45c315a226b8cfd [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:32:27.484989 22543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01d34370795342edb45c315a226b8cfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01d34370795342edb45c315a226b8cfd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33429 } }
I20260927 21:32:27.485688 22543 leader_election.cc:302] T 00000000000000000000000000000000 P 01d34370795342edb45c315a226b8cfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01d34370795342edb45c315a226b8cfd; no voters:
I20260927 21:32:27.486425 22543 leader_election.cc:288] T 00000000000000000000000000000000 P 01d34370795342edb45c315a226b8cfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:32:27.486558 22552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01d34370795342edb45c315a226b8cfd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:32:27.488547 22543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01d34370795342edb45c315a226b8cfd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:32:27.494074 22552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01d34370795342edb45c315a226b8cfd [term 1 LEADER]: Becoming Leader. State: Replica: 01d34370795342edb45c315a226b8cfd, State: Running, Role: LEADER
I20260927 21:32:27.494771 22552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01d34370795342edb45c315a226b8cfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01d34370795342edb45c315a226b8cfd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33429 } }
I20260927 21:32:27.497821 22554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01d34370795342edb45c315a226b8cfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01d34370795342edb45c315a226b8cfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01d34370795342edb45c315a226b8cfd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33429 } } }
I20260927 21:32:27.497795 22555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01d34370795342edb45c315a226b8cfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01d34370795342edb45c315a226b8cfd. Latest consensus state: current_term: 1 leader_uuid: "01d34370795342edb45c315a226b8cfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01d34370795342edb45c315a226b8cfd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33429 } } }
I20260927 21:32:27.498277 22555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01d34370795342edb45c315a226b8cfd [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:27.498277 22554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01d34370795342edb45c315a226b8cfd [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:27.499153 22563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:32:27.513051 22563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:32:27.515384 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:27.486065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58637 (local address 127.0.97.62:33429)
0927 21:32:27.486303 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:27.486311 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:27.493304 (+ 6993us) server_negotiation.cc:407] Connection header received
0927 21:32:27.493559 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:27.493567 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:27.493724 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:27.493881 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:27.496789 (+ 2908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:27.498044 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:27.500543 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:27.501551 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:27.501722 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:27.506078 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:27.506084 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:27.506092 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:27.506147 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:27.510974 (+ 4827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:27.512049 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:27.512058 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:27.512064 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:27.512249 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:27.512815 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:27.512823 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:27.512829 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:27.513348 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:27.513654 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:27.514434 (+ 780us) server_negotiation.cc:299] Negotiation successful
0927 21:32:27.514785 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:32:27.524022 22563 catalog_manager.cc:1347] Generated new cluster ID: cf971cb9f5464ab88a3e35030bc93d66
I20260927 21:32:27.524152 22563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:32:27.544301 22563 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:32:27.545938 22563 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:32:27.553114 22563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01d34370795342edb45c315a226b8cfd: Generated new TSK 0
I20260927 21:32:27.554196 22563 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:32:27.564230 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 22476
I20260927 21:32:27.565433 22538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:32:27.659281 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 22476
2026-09-27T21:32:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2012 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:32:27 dist-test-slave-bxm0 krb5kdc[22575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:32:27 dist-test-slave-bxm0 krb5kdc[22575](info): set up 2 sockets
Sep 27 21:32:27 dist-test-slave-bxm0 krb5kdc[22575](info): commencing operation
krb5kdc: starting...
W20260927 21:32:28.778031 388 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:32:28 dist-test-slave-bxm0 krb5kdc[22575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:32:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:32:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:32:29.032609 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42181
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:32:29.361210 22588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:29.361495 22588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:29.361709 22588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:29.369012 22588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:32:29.369123 22588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:29.369248 22588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:29.369347 22588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:32:29.369429 22588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:32:29.379320 22588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:29.380645 22588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:29.381947 22588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:29.394021 22593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:29.393962 22594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:29.393962 22596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:29.394734 22588 server_base.cc:1034] running on GCE node
I20260927 21:32:29.395517 22588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:29.396905 22588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:29.398109 22588 hybrid_clock.cc:648] HybridClock initialized: now 1790544749398059 us; error 39 us; skew 500 ppm
I20260927 21:32:29.398476 22588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:29 dist-test-slave-bxm0 krb5kdc[22575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544749, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:29.401495 22588 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:32:29.402974 22588 webserver.cc:460] Webserver started at http://127.0.97.62:40107/ using document root <none> and password file <none>
I20260927 21:32:29.403652 22588 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:29.403797 22588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:29.404102 22588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:29.407104 22588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "d457b088e3644cddba7a40b56bc824e2"
format_stamp: "Formatted at 2026-09-27 21:32:29 on dist-test-slave-bxm0"
server_key: "4bb6381d5c09ff9b218027dc31c03409"
server_key_iv: "bb4db88148ce5ba593da443fb6c977d9"
server_key_version: "encryptionkey@0"
I20260927 21:32:29.407858 22588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "d457b088e3644cddba7a40b56bc824e2"
format_stamp: "Formatted at 2026-09-27 21:32:29 on dist-test-slave-bxm0"
server_key: "4bb6381d5c09ff9b218027dc31c03409"
server_key_iv: "bb4db88148ce5ba593da443fb6c977d9"
server_key_version: "encryptionkey@0"
I20260927 21:32:29.412998 22588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260927 21:32:29.416553 22603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:29.417959 22588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 21:32:29.418154 22588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "d457b088e3644cddba7a40b56bc824e2"
format_stamp: "Formatted at 2026-09-27 21:32:29 on dist-test-slave-bxm0"
server_key: "4bb6381d5c09ff9b218027dc31c03409"
server_key_iv: "bb4db88148ce5ba593da443fb6c977d9"
server_key_version: "encryptionkey@0"
I20260927 21:32:29.418360 22588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:29.450287 22588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:29.454493 22588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:29.454798 22588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:29.474025 22588 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42181
I20260927 21:32:29.474062 22654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42181 every 8 connection(s)
I20260927 21:32:29.475479 22588 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:32:29.478278 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 22588
I20260927 21:32:29.478463 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:32:29.478875 388 external_mini_cluster.cc:1428] Setting key 619c12377623d5b10baa0df61bea1e23
I20260927 21:32:29.479503 22655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:32:29.491825 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:29.480657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45271 (local address 127.0.97.62:42181)
0927 21:32:29.481426 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:29.481438 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:29.481476 (+ 38us) server_negotiation.cc:407] Connection header received
0927 21:32:29.483087 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:29.483109 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:29.483502 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:29.483916 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:29.484445 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:29.485881 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:29.487253 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:29.487926 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:29.488126 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:29.489403 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:29.489411 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:29.489426 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:32:29.490076 (+ 650us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:29.490167 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:32:29.490193 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:29.490337 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:29.490550 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 21:32:29.490821 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":333,"threads_started":1}
I20260927 21:32:29.492265 22655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d457b088e3644cddba7a40b56bc824e2: Bootstrap starting.
I20260927 21:32:29.495568 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:32:29.495846 22655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d457b088e3644cddba7a40b56bc824e2: Neither blocks nor log segments found. Creating new log.
I20260927 21:32:29.496800 22655 log.cc:824] T 00000000000000000000000000000000 P d457b088e3644cddba7a40b56bc824e2: Log is configured to *not* fsync() on all Append() calls
I20260927 21:32:29.499543 22655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d457b088e3644cddba7a40b56bc824e2: No bootstrap required, opened a new log
I20260927 21:32:29.506582 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:29.500042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57503 (local address 127.0.97.62:42181)
0927 21:32:29.500221 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:29.500227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:29.500612 (+ 385us) server_negotiation.cc:407] Connection header received
0927 21:32:29.500756 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:29.500761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:29.500842 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:29.500954 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:29.501438 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:29.502636 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:29.503922 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:29.504620 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:29.504757 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:29.505838 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:29.505845 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:29.505849 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:32:29.505871 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:29.505919 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:32:29.505944 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:29.506043 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:29.506131 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 21:32:29.506229 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 21:32:29.509234 22655 raft_consensus.cc:348] T 00000000000000000000000000000000 P d457b088e3644cddba7a40b56bc824e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d457b088e3644cddba7a40b56bc824e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42181 } }
I20260927 21:32:29.509796 22655 raft_consensus.cc:374] T 00000000000000000000000000000000 P d457b088e3644cddba7a40b56bc824e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:32:29.509928 22655 raft_consensus.cc:729] T 00000000000000000000000000000000 P d457b088e3644cddba7a40b56bc824e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d457b088e3644cddba7a40b56bc824e2, State: Initialized, Role: FOLLOWER
I20260927 21:32:29.510540 22655 consensus_queue.cc:260] T 00000000000000000000000000000000 P d457b088e3644cddba7a40b56bc824e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d457b088e3644cddba7a40b56bc824e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42181 } }
I20260927 21:32:29.510771 22655 raft_consensus.cc:388] T 00000000000000000000000000000000 P d457b088e3644cddba7a40b56bc824e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:32:29.510926 22655 raft_consensus.cc:482] T 00000000000000000000000000000000 P d457b088e3644cddba7a40b56bc824e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:32:29.511099 22655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d457b088e3644cddba7a40b56bc824e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:32:29.513278 22655 raft_consensus.cc:504] T 00000000000000000000000000000000 P d457b088e3644cddba7a40b56bc824e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d457b088e3644cddba7a40b56bc824e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42181 } }
I20260927 21:32:29.513852 22655 leader_election.cc:302] T 00000000000000000000000000000000 P d457b088e3644cddba7a40b56bc824e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d457b088e3644cddba7a40b56bc824e2; no voters:
I20260927 21:32:29.514431 22655 leader_election.cc:288] T 00000000000000000000000000000000 P d457b088e3644cddba7a40b56bc824e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:32:29.514595 22665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d457b088e3644cddba7a40b56bc824e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:32:29.515676 22665 raft_consensus.cc:686] T 00000000000000000000000000000000 P d457b088e3644cddba7a40b56bc824e2 [term 1 LEADER]: Becoming Leader. State: Replica: d457b088e3644cddba7a40b56bc824e2, State: Running, Role: LEADER
I20260927 21:32:29.516288 22665 consensus_queue.cc:237] T 00000000000000000000000000000000 P d457b088e3644cddba7a40b56bc824e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d457b088e3644cddba7a40b56bc824e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42181 } }
I20260927 21:32:29.516687 22655 sys_catalog.cc:564] T 00000000000000000000000000000000 P d457b088e3644cddba7a40b56bc824e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:32:29.519106 22666 sys_catalog.cc:455] T 00000000000000000000000000000000 P d457b088e3644cddba7a40b56bc824e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d457b088e3644cddba7a40b56bc824e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d457b088e3644cddba7a40b56bc824e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42181 } } }
I20260927 21:32:29.519706 22666 sys_catalog.cc:458] T 00000000000000000000000000000000 P d457b088e3644cddba7a40b56bc824e2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:29.519349 22667 sys_catalog.cc:455] T 00000000000000000000000000000000 P d457b088e3644cddba7a40b56bc824e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d457b088e3644cddba7a40b56bc824e2. Latest consensus state: current_term: 1 leader_uuid: "d457b088e3644cddba7a40b56bc824e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d457b088e3644cddba7a40b56bc824e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42181 } } }
I20260927 21:32:29.519838 22667 sys_catalog.cc:458] T 00000000000000000000000000000000 P d457b088e3644cddba7a40b56bc824e2 [sys.catalog]: This master's current role is: LEADER
W20260927 21:32:29.527333 22681 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d457b088e3644cddba7a40b56bc824e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:32:29.527474 22681 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:32:29.527623 22674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:32:29.529765 22674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:32:29.538269 22674 catalog_manager.cc:1347] Generated new cluster ID: 4f823ecd356346a3b7d4035925c3f2d7
I20260927 21:32:29.538383 22674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:32:29.568694 22674 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:32:29.570550 22674 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:32:29.580331 22674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d457b088e3644cddba7a40b56bc824e2: Generated new TSK 0
I20260927 21:32:29.581406 22674 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:32:29.598989 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 22588
W20260927 21:32:29.600093 22650 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45271
W20260927 21:32:29.600850 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 22588: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:32:29.600991 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 22588
2026-09-27T21:32:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1939 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:32:29 dist-test-slave-bxm0 krb5kdc[22686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:32:29 dist-test-slave-bxm0 krb5kdc[22686](info): set up 2 sockets
Sep 27 21:32:29 dist-test-slave-bxm0 krb5kdc[22686](info): commencing operation
krb5kdc: starting...
W20260927 21:32:30.710124 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:32:30 dist-test-slave-bxm0 krb5kdc[22686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:32:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:32:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:32:30.967123 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36911
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:32:31.295383 22699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:31.295679 22699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:31.295773 22699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:31.304379 22699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:32:31.304490 22699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:31.304538 22699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:31.304598 22699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:32:31.304641 22699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:32:31.314476 22699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:31.315682 22699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:31.316890 22699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:31.328745 22707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:31.328850 22705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:31.328874 22704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:31.329876 22699 server_base.cc:1034] running on GCE node
I20260927 21:32:31.330616 22699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:31.332010 22699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:31.333210 22699 hybrid_clock.cc:648] HybridClock initialized: now 1790544751333150 us; error 52 us; skew 500 ppm
I20260927 21:32:31.333547 22699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:31 dist-test-slave-bxm0 krb5kdc[22686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:31.336585 22699 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:32:31.338102 22699 webserver.cc:460] Webserver started at http://127.0.97.62:38957/ using document root <none> and password file <none>
I20260927 21:32:31.338783 22699 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:31.338912 22699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:31.339227 22699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:31.342242 22699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "25414a1b2cde492ea8369f4ddbebcd18"
format_stamp: "Formatted at 2026-09-27 21:32:31 on dist-test-slave-bxm0"
server_key: "75a99b7020d6bf40f53ba6fa66b24ed8"
server_key_iv: "432c4f731ee33966204f9697546a1a2e"
server_key_version: "encryptionkey@0"
I20260927 21:32:31.342986 22699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "25414a1b2cde492ea8369f4ddbebcd18"
format_stamp: "Formatted at 2026-09-27 21:32:31 on dist-test-slave-bxm0"
server_key: "75a99b7020d6bf40f53ba6fa66b24ed8"
server_key_iv: "432c4f731ee33966204f9697546a1a2e"
server_key_version: "encryptionkey@0"
I20260927 21:32:31.347995 22699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 21:32:31.351400 22714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:31.352742 22699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:32:31.352936 22699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "25414a1b2cde492ea8369f4ddbebcd18"
format_stamp: "Formatted at 2026-09-27 21:32:31 on dist-test-slave-bxm0"
server_key: "75a99b7020d6bf40f53ba6fa66b24ed8"
server_key_iv: "432c4f731ee33966204f9697546a1a2e"
server_key_version: "encryptionkey@0"
I20260927 21:32:31.353149 22699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:31.358927 22699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:31.362843 22699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:31.363142 22699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:31.381094 22699 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36911
I20260927 21:32:31.381138 22765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36911 every 8 connection(s)
I20260927 21:32:31.382555 22699 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:32:31.386293 22766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:32:31.392182 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 22699
I20260927 21:32:31.392361 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:32:31.392817 388 external_mini_cluster.cc:1428] Setting key 5f83b15a0afc956adf118cd04c9864f2
I20260927 21:32:31.397810 22766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25414a1b2cde492ea8369f4ddbebcd18: Bootstrap starting.
I20260927 21:32:31.401814 22766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25414a1b2cde492ea8369f4ddbebcd18: Neither blocks nor log segments found. Creating new log.
I20260927 21:32:31.401902 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:31.394745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42269 (local address 127.0.97.62:36911)
0927 21:32:31.395591 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:31.395606 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:31.395655 (+ 49us) server_negotiation.cc:407] Connection header received
0927 21:32:31.398049 (+ 2394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:31.398084 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:31.398534 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:31.398945 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:31.399989 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:31.399997 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:31.400014 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:32:31.400044 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:31.400121 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:32:31.400149 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:31.400364 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:31.400485 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 21:32:31.400669 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":356,"threads_started":1}
I20260927 21:32:31.403002 22766 log.cc:824] T 00000000000000000000000000000000 P 25414a1b2cde492ea8369f4ddbebcd18: Log is configured to *not* fsync() on all Append() calls
I20260927 21:32:31.405706 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:32:31.406173 22766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25414a1b2cde492ea8369f4ddbebcd18: No bootstrap required, opened a new log
I20260927 21:32:31.412484 22769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:31.410073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57111 (local address 127.0.97.62:36911)
0927 21:32:31.410268 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:31.410273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:31.410638 (+ 365us) server_negotiation.cc:407] Connection header received
0927 21:32:31.410764 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:31.410769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:31.410839 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:31.410948 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:31.411812 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:31.411819 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:31.411830 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:32:31.411851 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:31.411902 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:32:31.411921 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:31.412048 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:31.412119 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 21:32:31.412185 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 21:32:31.415023 22766 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25414a1b2cde492ea8369f4ddbebcd18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25414a1b2cde492ea8369f4ddbebcd18" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36911 } }
I20260927 21:32:31.415369 22766 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25414a1b2cde492ea8369f4ddbebcd18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:32:31.415447 22766 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25414a1b2cde492ea8369f4ddbebcd18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25414a1b2cde492ea8369f4ddbebcd18, State: Initialized, Role: FOLLOWER
I20260927 21:32:31.415941 22766 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25414a1b2cde492ea8369f4ddbebcd18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25414a1b2cde492ea8369f4ddbebcd18" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36911 } }
I20260927 21:32:31.416141 22766 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25414a1b2cde492ea8369f4ddbebcd18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:32:31.416239 22766 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25414a1b2cde492ea8369f4ddbebcd18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:32:31.416406 22766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25414a1b2cde492ea8369f4ddbebcd18 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:32:31.418824 22766 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25414a1b2cde492ea8369f4ddbebcd18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25414a1b2cde492ea8369f4ddbebcd18" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36911 } }
I20260927 21:32:31.419314 22766 leader_election.cc:302] T 00000000000000000000000000000000 P 25414a1b2cde492ea8369f4ddbebcd18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25414a1b2cde492ea8369f4ddbebcd18; no voters:
I20260927 21:32:31.419955 22766 leader_election.cc:288] T 00000000000000000000000000000000 P 25414a1b2cde492ea8369f4ddbebcd18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:32:31.420085 22776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25414a1b2cde492ea8369f4ddbebcd18 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:32:31.420944 22776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25414a1b2cde492ea8369f4ddbebcd18 [term 1 LEADER]: Becoming Leader. State: Replica: 25414a1b2cde492ea8369f4ddbebcd18, State: Running, Role: LEADER
I20260927 21:32:31.421485 22776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25414a1b2cde492ea8369f4ddbebcd18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25414a1b2cde492ea8369f4ddbebcd18" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36911 } }
I20260927 21:32:31.422215 22766 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25414a1b2cde492ea8369f4ddbebcd18 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:32:31.424576 22778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25414a1b2cde492ea8369f4ddbebcd18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25414a1b2cde492ea8369f4ddbebcd18. Latest consensus state: current_term: 1 leader_uuid: "25414a1b2cde492ea8369f4ddbebcd18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25414a1b2cde492ea8369f4ddbebcd18" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36911 } } }
I20260927 21:32:31.424661 22777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25414a1b2cde492ea8369f4ddbebcd18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25414a1b2cde492ea8369f4ddbebcd18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25414a1b2cde492ea8369f4ddbebcd18" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36911 } } }
I20260927 21:32:31.424958 22778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25414a1b2cde492ea8369f4ddbebcd18 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:31.425037 22777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25414a1b2cde492ea8369f4ddbebcd18 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:31.426970 22782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:32:31.433364 22782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:32:31.443629 22782 catalog_manager.cc:1347] Generated new cluster ID: d621635b95dc4bd4a443a9c16a482c94
I20260927 21:32:31.443753 22782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:32:31.465449 22782 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:32:31.465917 22782 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:32:31.474074 22782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25414a1b2cde492ea8369f4ddbebcd18: Generated new TSK 0
I20260927 21:32:31.474973 22782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:32:31.480597 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 22699
W20260927 21:32:31.482188 22761 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42269
W20260927 21:32:31.482923 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 22699: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 21:32:31.483073 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 22699
2026-09-27T21:32:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1881 ms)
[----------] 18 tests from AuthTokenIssuingTest (35225 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:32:31 dist-test-slave-bxm0 krb5kdc[22797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:32:31 dist-test-slave-bxm0 krb5kdc[22797](info): set up 2 sockets
Sep 27 21:32:31 dist-test-slave-bxm0 krb5kdc[22797](info): commencing operation
krb5kdc: starting...
W20260927 21:32:32.586124 388 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:32:32 dist-test-slave-bxm0 krb5kdc[22797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:32:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:32:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:32:32.848979 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41659
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41659,127.0.97.61:42509,127.0.97.60:38809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:32:33.181972 22810 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:32:33.182367 22810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:33.182468 22810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:33.182627 22810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:33.189956 22810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:32:33.190059 22810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:33.190158 22810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:33.190234 22810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:32:33.190326 22810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:32:33.200174 22810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41659,127.0.97.61:42509,127.0.97.60:38809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:33.201539 22810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:33.202901 22810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:33.214669 22815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:33.214594 22818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:33.214620 22816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:33.215601 22810 server_base.cc:1034] running on GCE node
I20260927 21:32:33.216477 22810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:33.217923 22810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:33.219153 22810 hybrid_clock.cc:648] HybridClock initialized: now 1790544753219089 us; error 37 us; skew 500 ppm
I20260927 21:32:33.219493 22810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:33 dist-test-slave-bxm0 krb5kdc[22797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:33.226496 22810 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:32:33.228088 22810 webserver.cc:460] Webserver started at http://127.0.97.62:42619/ using document root <none> and password file <none>
I20260927 21:32:33.228828 22810 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:33.228948 22810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:33.229295 22810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:33.232372 22810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "b276cf1185cf4410a917fadef3a52132"
format_stamp: "Formatted at 2026-09-27 21:32:33 on dist-test-slave-bxm0"
server_key: "7d14c0405c8b0996de34dca83c5a9e49"
server_key_iv: "b41fc969258b1e5a7a62ab8008d186a8"
server_key_version: "encryptionkey@0"
I20260927 21:32:33.233191 22810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "b276cf1185cf4410a917fadef3a52132"
format_stamp: "Formatted at 2026-09-27 21:32:33 on dist-test-slave-bxm0"
server_key: "7d14c0405c8b0996de34dca83c5a9e49"
server_key_iv: "b41fc969258b1e5a7a62ab8008d186a8"
server_key_version: "encryptionkey@0"
I20260927 21:32:33.238555 22810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 21:32:33.242004 22825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:33.243278 22810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:32:33.243463 22810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "b276cf1185cf4410a917fadef3a52132"
format_stamp: "Formatted at 2026-09-27 21:32:33 on dist-test-slave-bxm0"
server_key: "7d14c0405c8b0996de34dca83c5a9e49"
server_key_iv: "b41fc969258b1e5a7a62ab8008d186a8"
server_key_version: "encryptionkey@0"
I20260927 21:32:33.243676 22810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:33.260783 22810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:33.265151 22810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:33.265445 22810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:33.284534 22810 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41659
I20260927 21:32:33.284552 22876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41659 every 8 connection(s)
I20260927 21:32:33.286018 22810 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:32:33.289774 22877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:32:33.294293 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 22810
I20260927 21:32:33.294533 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:32:33.295373 388 external_mini_cluster.cc:1428] Setting key 573eea6a76a123bcf41ef6821670b463
I20260927 21:32:33.297865 22877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41659 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:32:33 dist-test-slave-bxm0 krb5kdc[22797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260927 21:32:33.326063 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:33.303311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44128 (local address 127.0.97.62:41659)
0927 21:32:33.304103 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:33.304113 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:33.304186 (+ 73us) server_negotiation.cc:407] Connection header received
0927 21:32:33.306479 (+ 2293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:33.306495 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:33.306943 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:33.307179 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:33.308901 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:33.310831 (+ 1930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:33.312278 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:33.313282 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:33.313492 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:33.317193 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:33.317206 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:33.317226 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:33.317271 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:33.321449 (+ 4178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:33.322234 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:33.322243 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:33.322247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:33.322461 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:33.322896 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:33.322904 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:33.322908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:33.323649 (+ 741us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:33.323867 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:33.324489 (+ 622us) server_negotiation.cc:299] Negotiation successful
0927 21:32:33.324955 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":154,"server-negotiator.queue_time_us":567,"thread_start_us":398,"threads_started":1}
I20260927 21:32:33.326073 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:33.301958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41659 (local address 127.0.0.1:44128)
0927 21:32:33.302840 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:33.302894 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 21:32:33.305295 (+ 2401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:32:33.307207 (+ 1912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:32:33.307215 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:32:33.308035 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:32:33.308608 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:33.308639 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:33.311004 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:33.311008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:33.312109 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:33.312122 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:33.313507 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:33.313511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:33.313797 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:33.314486 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:32:33.314509 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:32:33.316935 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:32:33.321604 (+ 4669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:33.321662 (+ 58us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:33.321683 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:33.322087 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:33.322585 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:33.322588 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:33.322590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:33.322761 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:33.323857 (+ 1096us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:32:33.323864 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:32:33.324061 (+ 197us) client_negotiation.cc:769] Sending connection context
0927 21:32:33.324290 (+ 229us) client_negotiation.cc:240] Negotiation successful
0927 21:32:33.324702 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":21,"thread_start_us":385,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260927 21:32:33.329371 22877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:32:33.330551 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:33.330167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:40234)
0927 21:32:33.330330 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:33.330434 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 21:32:33.330857 22828 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
I20260927 21:32:33.332852 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:33.332377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:40246)
0927 21:32:33.332637 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:33.332728 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260927 21:32:33.333058 22877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42509: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260927 21:32:33.359059 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42509
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41659,127.0.97.61:42509,127.0.97.60:38809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:32:33.382401 22877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attempt: 1
I20260927 21:32:33.383693 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:33.383180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:40260)
0927 21:32:33.383403 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:33.383527 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 21:32:33.384696 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:33.384313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:40274)
0927 21:32:33.384457 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:33.384535 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 21:32:33.384912 22877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42509: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
I20260927 21:32:33.453219 22877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attempt: 2
I20260927 21:32:33.454495 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:33.454027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:40278)
0927 21:32:33.454269 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:33.454362 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 21:32:33.455703 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:33.455351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:40290)
0927 21:32:33.455477 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:33.455557 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 21:32:33.455978 22877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42509: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
I20260927 21:32:33.547258 22877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attempt: 3
I20260927 21:32:33.548457 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:33.548025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:40300)
0927 21:32:33.548235 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:33.548326 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 21:32:33.549505 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:33.549124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:40312)
0927 21:32:33.549289 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:33.549369 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 21:32:33.549743 22877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42509: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
W20260927 21:32:33.689796 22884 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:32:33.690182 22884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:33.690310 22884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:33.690448 22884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 21:32:33.692965 22877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attempt: 4
I20260927 21:32:33.694145 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:33.693569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:40318)
0927 21:32:33.693935 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:33.694021 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205}
I20260927 21:32:33.695098 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:33.694771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:40326)
0927 21:32:33.694910 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:33.694990 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 21:32:33.695281 22877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42509: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
W20260927 21:32:33.697945 22884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:32:33.698064 22884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:33.698200 22884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:33.698287 22884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:32:33.698395 22884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:32:33.708294 22884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41659,127.0.97.61:42509,127.0.97.60:38809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:33.709663 22884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:33.710963 22884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:33.722759 22892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:33.722889 22889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:33.722859 22890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:33.723766 22884 server_base.cc:1034] running on GCE node
I20260927 21:32:33.724560 22884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:33.726132 22884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:33.727522 22884 hybrid_clock.cc:648] HybridClock initialized: now 1790544753727420 us; error 70 us; skew 500 ppm
I20260927 21:32:33.727881 22884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:33 dist-test-slave-bxm0 krb5kdc[22797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:33.731637 22884 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260927 21:32:33.733174 22884 webserver.cc:460] Webserver started at http://127.0.97.61:39375/ using document root <none> and password file <none>
I20260927 21:32:33.733952 22884 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:33.734069 22884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:33.734393 22884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:33.737357 22884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-1/data/instance:
uuid: "4e91409a5a9e4439a7fad9d75badefb8"
format_stamp: "Formatted at 2026-09-27 21:32:33 on dist-test-slave-bxm0"
server_key: "6efb077672d2cea782b655d6e834ad88"
server_key_iv: "9d5bd22f3b6504d2d14870b734a0be9d"
server_key_version: "encryptionkey@0"
I20260927 21:32:33.738207 22884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-1/wal/instance:
uuid: "4e91409a5a9e4439a7fad9d75badefb8"
format_stamp: "Formatted at 2026-09-27 21:32:33 on dist-test-slave-bxm0"
server_key: "6efb077672d2cea782b655d6e834ad88"
server_key_iv: "9d5bd22f3b6504d2d14870b734a0be9d"
server_key_version: "encryptionkey@0"
I20260927 21:32:33.743198 22884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 21:32:33.747045 22899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:33.748935 22884 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 21:32:33.749081 22884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-1/wal
uuid: "4e91409a5a9e4439a7fad9d75badefb8"
format_stamp: "Formatted at 2026-09-27 21:32:33 on dist-test-slave-bxm0"
server_key: "6efb077672d2cea782b655d6e834ad88"
server_key_iv: "9d5bd22f3b6504d2d14870b734a0be9d"
server_key_version: "encryptionkey@0"
I20260927 21:32:33.749264 22884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:33.770275 22884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:33.774307 22884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:33.774526 22884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:33.793073 22884 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:42509
I20260927 21:32:33.793116 22950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:42509 every 8 connection(s)
I20260927 21:32:33.794607 22884 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-1/data/info.pb
I20260927 21:32:33.798589 22951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:32:33.804863 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 22884
I20260927 21:32:33.805047 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-1/wal/instance
I20260927 21:32:33.805477 388 external_mini_cluster.cc:1428] Setting key 44d12d5c58f8e48da89c7ffcc21e87a2
I20260927 21:32:33.805053 22951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41659 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:32:33 dist-test-slave-bxm0 krb5kdc[22797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:32:33.830263 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:33.807727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44132 (local address 127.0.97.62:41659)
0927 21:32:33.807964 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:33.807973 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:33.809116 (+ 1143us) server_negotiation.cc:407] Connection header received
0927 21:32:33.812099 (+ 2983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:33.812104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:33.812172 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:33.812261 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:33.814424 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:33.815615 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:33.817177 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:33.818055 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:33.818234 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:33.822211 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:33.822221 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:33.822229 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:33.822283 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:33.826340 (+ 4057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:33.827202 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:33.827212 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:33.827219 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:33.827403 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:33.827866 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:33.827874 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:33.827880 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:33.828351 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:33.828539 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:33.829380 (+ 841us) server_negotiation.cc:299] Negotiation successful
0927 21:32:33.829720 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 21:32:33.830885 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:33.808129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41659 (local address 127.0.0.1:44132)
0927 21:32:33.808923 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:33.808979 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 21:32:33.811742 (+ 2763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:32:33.812538 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:32:33.812552 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:32:33.813344 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:32:33.814203 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:33.814231 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:33.815802 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:33.815806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:33.817001 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:33.817013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:33.818380 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:33.818385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:33.818590 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:33.819368 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:32:33.819391 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:32:33.821934 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:32:33.826535 (+ 4601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:33.826543 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:33.826559 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:33.827029 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:33.827534 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:33.827537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:33.827539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:33.827729 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:33.828510 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:32:33.828521 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:32:33.828940 (+ 419us) client_negotiation.cc:769] Sending connection context
0927 21:32:33.829199 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 21:32:33.829666 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":297,"threads_started":1}
I20260927 21:32:33.832912 22951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 27 21:32:33 dist-test-slave-bxm0 krb5kdc[22797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 21:32:33.849267 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:33.833891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:40330)
0927 21:32:33.834137 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:33.834155 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:32:33.834254 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:32:33.835238 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:32:33.835242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:32:33.835700 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:32:33.835851 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:33.835860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:33.837744 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:33.837748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:33.838619 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:33.838631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:33.839908 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:33.839913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:33.840070 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:33.840769 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:32:33.840788 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:32:33.843050 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:32:33.846766 (+ 3716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:33.846774 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:33.846778 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:33.847143 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:33.847515 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:33.847522 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:33.847525 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:33.847646 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:33.848243 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:32:33.848252 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:32:33.848427 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 21:32:33.848562 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 21:32:33.848871 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 21:32:33.849542 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:33.834152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40330 (local address 127.0.97.61:42509)
0927 21:32:33.834819 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:33.834825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:33.834851 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:32:33.834894 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:33.834900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:33.835079 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:33.835202 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:33.836011 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:33.837514 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:33.838765 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:33.839722 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:33.839871 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:33.843284 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:33.843301 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:33.843322 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:33.843371 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:33.846632 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:33.847287 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:33.847294 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:33.847297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:33.847409 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:33.847762 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:33.847767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:33.847770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:33.848110 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:33.848342 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:33.848721 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 21:32:33.848987 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":509,"thread_start_us":369,"threads_started":1}
I20260927 21:32:33.852176 22951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:32:33.853250 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:33.852794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38809 (local address 127.0.0.1:35852)
0927 21:32:33.852973 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:33.853091 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 21:32:33.853490 22900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
I20260927 21:32:33.855132 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:33.854789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38809 (local address 127.0.0.1:35854)
0927 21:32:33.854930 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:33.855028 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 21:32:33.855371 22951 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38809: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260927 21:32:33.868975 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:38809
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41659,127.0.97.61:42509,127.0.97.60:38809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:32:33.904722 22951 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } attempt: 1
I20260927 21:32:33.905972 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:33.905495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38809 (local address 127.0.0.1:35856)
0927 21:32:33.905734 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:33.905820 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 21:32:33.906970 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:33.906615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38809 (local address 127.0.0.1:35872)
0927 21:32:33.906778 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:33.906869 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 21:32:33.907186 22951 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38809: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
I20260927 21:32:33.975418 22951 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } attempt: 2
I20260927 21:32:33.976549 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:33.976072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38809 (local address 127.0.0.1:35876)
0927 21:32:33.976302 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:33.976396 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 21:32:33.977674 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:33.977203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38809 (local address 127.0.0.1:35878)
0927 21:32:33.977395 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:33.977480 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 21:32:33.977878 22951 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38809: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
I20260927 21:32:33.994534 22877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attempt: 5
Sep 27 21:32:33 dist-test-slave-bxm0 krb5kdc[22797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 21:32:34.008250 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:33.995345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:40338)
0927 21:32:33.995568 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:33.995592 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 21:32:33.995787 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:32:33.996138 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:32:33.996145 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:32:33.996581 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:32:33.996764 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:33.996772 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:33.997869 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:33.997872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:33.998726 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:33.998737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:33.999495 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:33.999501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:33.999703 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:34.000456 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:32:34.000475 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:32:34.002616 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:32:34.005766 (+ 3150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:34.005772 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:34.005776 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:34.006168 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:34.006556 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:34.006561 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:34.006565 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:34.006713 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:34.007262 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:32:34.007270 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:32:34.007431 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 21:32:34.007586 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 21:32:34.007894 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 21:32:34.008251 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:33.995409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40338 (local address 127.0.97.61:42509)
0927 21:32:33.995598 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:33.995604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:33.995703 (+ 99us) server_negotiation.cc:407] Connection header received
0927 21:32:33.995918 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:33.995925 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:33.995992 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:33.996099 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:33.996918 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:33.997683 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:33.998881 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:33.999334 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:33.999467 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:34.002825 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:34.002831 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:34.002834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:34.002867 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:34.005580 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:34.006291 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:34.006298 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:34.006302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:34.006441 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:34.006826 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:34.006832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:34.006836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:34.007111 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:34.007302 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:34.007691 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 21:32:34.007901 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 21:32:34.009747 22877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:32:34.010787 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.010408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38809 (local address 127.0.0.1:35892)
0927 21:32:34.010565 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:34.010639 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 21:32:34.011679 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.011358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38809 (local address 127.0.0.1:35906)
0927 21:32:34.011507 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:34.011583 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 21:32:34.011874 22877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38809: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
I20260927 21:32:34.063100 22877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } attempt: 1
I20260927 21:32:34.064359 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.063894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38809 (local address 127.0.0.1:35920)
0927 21:32:34.064106 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:34.064196 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 21:32:34.065367 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.065034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38809 (local address 127.0.0.1:35926)
0927 21:32:34.065184 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:34.065265 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 21:32:34.065656 22877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38809: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
I20260927 21:32:34.069106 22951 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } attempt: 3
I20260927 21:32:34.070150 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.069749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38809 (local address 127.0.0.1:35938)
0927 21:32:34.069936 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:34.070011 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 21:32:34.071141 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.070763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38809 (local address 127.0.0.1:35944)
0927 21:32:34.070980 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:34.071048 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260927 21:32:34.071350 22951 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38809: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
I20260927 21:32:34.133874 22877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } attempt: 2
I20260927 21:32:34.135004 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.134520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38809 (local address 127.0.0.1:35948)
0927 21:32:34.134753 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:34.134839 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 21:32:34.135953 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.135635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38809 (local address 127.0.0.1:35964)
0927 21:32:34.135780 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:34.135852 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 21:32:34.136157 22877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38809: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
W20260927 21:32:34.202323 22958 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:32:34.202729 22958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:34.202862 22958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:34.202999 22958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:34.210337 22958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:32:34.210465 22958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:34.210589 22958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:34.210676 22958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:32:34.210778 22958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:32:34.214531 22951 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } attempt: 4
I20260927 21:32:34.215647 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.215173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38809 (local address 127.0.0.1:35976)
0927 21:32:34.215404 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:34.215498 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 21:32:34.216723 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.216365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38809 (local address 127.0.0.1:35982)
0927 21:32:34.216541 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:34.216620 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 21:32:34.216967 22951 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38809: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
I20260927 21:32:34.221259 22958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41659,127.0.97.61:42509,127.0.97.60:38809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:38809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:34.222661 22958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:34.223989 22958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:34.235790 22963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:34.235843 22964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:34.235756 22966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:34.236979 22958 server_base.cc:1034] running on GCE node
I20260927 21:32:34.237807 22958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:34.239151 22958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:34.240430 22958 hybrid_clock.cc:648] HybridClock initialized: now 1790544754240351 us; error 43 us; skew 500 ppm
I20260927 21:32:34.240777 22958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:32:34.242367 22877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } attempt: 3
I20260927 21:32:34.243475 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.242920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38809 (local address 127.0.0.1:35998)
0927 21:32:34.243150 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:34.243252 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
Sep 27 21:32:34 dist-test-slave-bxm0 krb5kdc[22797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:34.244248 22958 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260927 21:32:34.244746 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.244233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38809 (local address 127.0.0.1:36012)
0927 21:32:34.244472 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:34.244576 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260927 21:32:34.244987 22877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38809: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38809: connect: Connection refused (error 111)
I20260927 21:32:34.245862 22958 webserver.cc:460] Webserver started at http://127.0.97.60:37579/ using document root <none> and password file <none>
I20260927 21:32:34.246549 22958 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:34.246675 22958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:34.246990 22958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:34.250068 22958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-2/data/instance:
uuid: "f60e2c1e7dd640d3a4732445af9d3a5e"
format_stamp: "Formatted at 2026-09-27 21:32:34 on dist-test-slave-bxm0"
server_key: "e4c4ba2efb6295c2dfe073a143fefec4"
server_key_iv: "29e9c31398d7f1a1a38246b37aab8eac"
server_key_version: "encryptionkey@0"
I20260927 21:32:34.250842 22958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-2/wal/instance:
uuid: "f60e2c1e7dd640d3a4732445af9d3a5e"
format_stamp: "Formatted at 2026-09-27 21:32:34 on dist-test-slave-bxm0"
server_key: "e4c4ba2efb6295c2dfe073a143fefec4"
server_key_iv: "29e9c31398d7f1a1a38246b37aab8eac"
server_key_version: "encryptionkey@0"
I20260927 21:32:34.256206 22958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 21:32:34.259680 22973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:34.261010 22958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 21:32:34.261215 22958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-2/wal
uuid: "f60e2c1e7dd640d3a4732445af9d3a5e"
format_stamp: "Formatted at 2026-09-27 21:32:34 on dist-test-slave-bxm0"
server_key: "e4c4ba2efb6295c2dfe073a143fefec4"
server_key_iv: "29e9c31398d7f1a1a38246b37aab8eac"
server_key_version: "encryptionkey@0"
I20260927 21:32:34.261441 22958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:34.292146 22958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:34.296990 22958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:34.297224 22958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:34.316381 22958 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:38809
I20260927 21:32:34.316401 23024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:38809 every 8 connection(s)
I20260927 21:32:34.317734 22958 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-2/data/info.pb
I20260927 21:32:34.321501 23025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:32:34.324050 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 22958
I20260927 21:32:34.324223 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790544390748963-388-0/minicluster-data/master-2/wal/instance
I20260927 21:32:34.324647 388 external_mini_cluster.cc:1428] Setting key ceee9004d148bfe8f5ca598b69d4d4ee
I20260927 21:32:34.329720 23025 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41659 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:32:34 dist-test-slave-bxm0 krb5kdc[22797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 21:32:34 dist-test-slave-bxm0 krb5kdc[22797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:32:34.346925 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.332732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44140 (local address 127.0.97.62:41659)
0927 21:32:34.333379 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:34.333389 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:34.334163 (+ 774us) server_negotiation.cc:407] Connection header received
0927 21:32:34.334418 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:34.334423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:34.334486 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:34.334603 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:34.335739 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.336527 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.337860 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.338257 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.338389 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:34.341547 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:34.341551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:34.341554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:34.341585 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:34.344199 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:34.344896 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:34.344901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:34.344906 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:34.345028 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:34.345402 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:34.345407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:34.345412 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:34.345720 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:34.345871 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:34.346308 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 21:32:34.346524 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":310,"threads_started":1}
I20260927 21:32:34.347765 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.326651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36022 (local address 127.0.97.60:38809)
0927 21:32:34.327444 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:34.327462 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:34.327512 (+ 50us) server_negotiation.cc:407] Connection header received
0927 21:32:34.329862 (+ 2350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:34.329893 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:34.330448 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:34.330965 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:34.332387 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.334149 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.335457 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.336104 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.336312 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:34.340431 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:34.340439 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:34.340450 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:34.340479 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:34.343540 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:34.344206 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:34.344218 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:34.344224 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:34.344365 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:34.344750 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:34.344755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:34.344760 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:34.345234 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:34.345523 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:34.346026 (+ 503us) server_negotiation.cc:299] Negotiation successful
0927 21:32:34.346648 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":551,"thread_start_us":328,"threads_started":1}
I20260927 21:32:34.347810 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.332770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41659 (local address 127.0.0.1:44140)
0927 21:32:34.334019 (+ 1249us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:34.334066 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:32:34.334225 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:32:34.334667 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:32:34.334675 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:32:34.335294 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:32:34.335585 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:34.335595 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.336675 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.336682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:34.337722 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:34.337734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.338480 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.338487 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:34.338631 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:34.339226 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:32:34.339258 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:32:34.341342 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:32:34.344330 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:34.344343 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:34.344350 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:34.344770 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:34.345139 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:34.345144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:34.345148 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:34.345285 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:34.345849 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:32:34.345857 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:32:34.346039 (+ 182us) client_negotiation.cc:769] Sending connection context
0927 21:32:34.346209 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 21:32:34.346557 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1009,"mutex_wait_us":22,"thread_start_us":874,"threads_started":1}
I20260927 21:32:34.349444 23025 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:32:34 dist-test-slave-bxm0 krb5kdc[22797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 27 21:32:34 dist-test-slave-bxm0 krb5kdc[22797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:32:34.364069 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.350191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:40352)
0927 21:32:34.350359 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:34.350378 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:32:34.350484 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:32:34.350800 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:32:34.350803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:32:34.351256 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:32:34.351440 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:34.351452 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.352877 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.352880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:34.354275 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:34.354291 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.355144 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.355151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:34.355370 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:34.356011 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:32:34.356031 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:32:34.358401 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:32:34.361665 (+ 3264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:34.361672 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:34.361676 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:34.362035 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:34.362417 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:34.362422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:34.362425 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:34.362570 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:34.363090 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:32:34.363095 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:32:34.363256 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 21:32:34.363392 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 21:32:34.363685 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 21:32:34.364096 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.350271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40352 (local address 127.0.97.61:42509)
0927 21:32:34.350469 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:34.350476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:34.350499 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:32:34.350588 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:34.350593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:34.350671 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:34.350776 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:34.351602 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.352725 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.354460 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.354981 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.355114 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:34.358593 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:34.358598 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:34.358602 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:34.358634 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:34.361507 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:34.362154 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:34.362161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:34.362164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:34.362301 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:34.362700 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:34.362703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:34.362705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:34.362971 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:34.363130 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:34.363511 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 21:32:34.363732 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:32:34.365345 23025 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:32:34.366245 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.351245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44148 (local address 127.0.97.62:41659)
0927 21:32:34.351437 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:34.351442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:34.351461 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:32:34.351562 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:34.351569 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:34.351652 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:34.351726 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:34.353155 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.354178 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.355797 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.356234 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.356366 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:34.360521 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:34.360525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:34.360528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:34.360558 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:34.363247 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:34.363892 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:34.363898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:34.363902 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:34.364021 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:34.364529 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:34.364534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:34.364538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:34.364871 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:34.365163 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:34.365645 (+ 482us) server_negotiation.cc:299] Negotiation successful
0927 21:32:34.365868 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 21:32:34 dist-test-slave-bxm0 krb5kdc[22797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 21:32:34 dist-test-slave-bxm0 krb5kdc[22797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 21:32:34.378187 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.366063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38809 (local address 127.0.0.1:36028)
0927 21:32:34.366224 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:34.366241 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:32:34.366334 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:32:34.366673 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:32:34.366678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:32:34.367035 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:32:34.367155 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:34.367194 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.368329 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.368334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:34.369241 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:34.369252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.370046 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.370052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:34.370240 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:34.370819 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:32:34.370836 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:32:34.372939 (+ 2103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:32:34.375745 (+ 2806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:34.375750 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:34.375754 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:34.376087 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:34.376491 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:34.376494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:34.376496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:34.376625 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:34.377245 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:32:34.377252 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:32:34.377422 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 21:32:34.377554 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 21:32:34.377875 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 21:32:34.378783 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.366294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36028 (local address 127.0.97.60:38809)
0927 21:32:34.366450 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:34.366455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:34.366471 (+ 16us) server_negotiation.cc:407] Connection header received
0927 21:32:34.366511 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:34.366514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:34.366560 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:34.366657 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:34.367329 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.368208 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.369378 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.369906 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.370002 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:34.373110 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:34.373114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:34.373117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:34.373144 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:34.375572 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:34.376219 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:34.376226 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:34.376232 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:34.376377 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:34.376742 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:34.376747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:34.376750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:34.377121 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:34.377259 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:34.377734 (+ 475us) server_negotiation.cc:299] Negotiation successful
0927 21:32:34.378025 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":64}
I20260927 21:32:34.383008 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.367830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40360 (local address 127.0.97.61:42509)
0927 21:32:34.367985 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:34.367990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:34.368007 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:32:34.368167 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:34.368172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:34.368239 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:34.368345 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:34.369510 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.370459 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.371693 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.372161 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.372256 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:34.376470 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:34.376478 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:34.376483 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:34.376524 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:34.380275 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:34.380976 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:34.380983 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:34.380987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:34.381146 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:34.381497 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:34.381501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:34.381505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:34.381869 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:34.382131 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:34.382430 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 21:32:34.382672 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:32:34.384555 23025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f60e2c1e7dd640d3a4732445af9d3a5e: Bootstrap starting.
I20260927 21:32:34.387748 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:32:34.388055 23025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f60e2c1e7dd640d3a4732445af9d3a5e: Neither blocks nor log segments found. Creating new log.
I20260927 21:32:34.389119 23025 log.cc:824] T 00000000000000000000000000000000 P f60e2c1e7dd640d3a4732445af9d3a5e: Log is configured to *not* fsync() on all Append() calls
I20260927 21:32:34.391811 23025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f60e2c1e7dd640d3a4732445af9d3a5e: No bootstrap required, opened a new log
I20260927 21:32:34.390628 22852 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b276cf1185cf4410a917fadef3a52132"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44148
I20260927 21:32:34.398952 23025 raft_consensus.cc:348] T 00000000000000000000000000000000 P f60e2c1e7dd640d3a4732445af9d3a5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276cf1185cf4410a917fadef3a52132" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41659 } } peers { permanent_uuid: "4e91409a5a9e4439a7fad9d75badefb8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "f60e2c1e7dd640d3a4732445af9d3a5e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } }
I20260927 21:32:34.399310 23025 raft_consensus.cc:729] T 00000000000000000000000000000000 P f60e2c1e7dd640d3a4732445af9d3a5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f60e2c1e7dd640d3a4732445af9d3a5e, State: Initialized, Role: FOLLOWER
I20260927 21:32:34.399758 23025 consensus_queue.cc:260] T 00000000000000000000000000000000 P f60e2c1e7dd640d3a4732445af9d3a5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276cf1185cf4410a917fadef3a52132" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41659 } } peers { permanent_uuid: "4e91409a5a9e4439a7fad9d75badefb8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "f60e2c1e7dd640d3a4732445af9d3a5e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } }
I20260927 21:32:34.400729 23032 sys_catalog.cc:455] T 00000000000000000000000000000000 P f60e2c1e7dd640d3a4732445af9d3a5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276cf1185cf4410a917fadef3a52132" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41659 } } peers { permanent_uuid: "4e91409a5a9e4439a7fad9d75badefb8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "f60e2c1e7dd640d3a4732445af9d3a5e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } } }
I20260927 21:32:34.401108 23032 sys_catalog.cc:458] T 00000000000000000000000000000000 P f60e2c1e7dd640d3a4732445af9d3a5e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:32:34.401958 23025 sys_catalog.cc:564] T 00000000000000000000000000000000 P f60e2c1e7dd640d3a4732445af9d3a5e [sys.catalog]: configured and running, proceeding with master startup.
W20260927 21:32:34.412119 23043 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f60e2c1e7dd640d3a4732445af9d3a5e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:32:34.412328 23043 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:32:34.409811 22958 master_runner.cc:186] Error getting master registration for 127.0.97.62:41659: OK, instance_id { permanent_uuid: "b276cf1185cf4410a917fadef3a52132" instance_seqno: 1790544753279111 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:32:34.421166 22958 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:32:34.422222 22877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } attempt: 4
Sep 27 21:32:34 dist-test-slave-bxm0 krb5kdc[22797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 21:32:34.438485 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.423221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36044 (local address 127.0.97.60:38809)
0927 21:32:34.423402 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:34.423406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:34.423426 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:32:34.423480 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:34.423483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:34.423555 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:34.423687 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:34.424529 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.425311 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.426723 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.427237 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.427392 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:34.431415 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:34.431421 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:34.431424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:34.431457 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:34.435336 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:34.436036 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:34.436044 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:34.436048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:34.436232 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:34.436667 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:34.436679 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:34.436685 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:34.437192 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:34.437353 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:34.437820 (+ 467us) server_negotiation.cc:299] Negotiation successful
0927 21:32:34.438054 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 21:32:34.438485 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.422987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38809 (local address 127.0.0.1:36044)
0927 21:32:34.423204 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:34.423226 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:32:34.423361 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:32:34.423722 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:32:34.423727 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:32:34.424175 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:32:34.424354 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:34.424363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.425474 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.425481 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:34.426552 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:34.426565 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.427403 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.427409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:34.427646 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:34.428335 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:32:34.428365 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:32:34.431206 (+ 2841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:32:34.435481 (+ 4275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:34.435488 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:34.435492 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:34.435887 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:34.436353 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:34.436359 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:34.436363 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:34.436542 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:34.437328 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:32:34.437333 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:32:34.437527 (+ 194us) client_negotiation.cc:769] Sending connection context
0927 21:32:34.437711 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 21:32:34.438057 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 21:32:34.444998 22877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132: Bootstrap starting.
I20260927 21:32:34.449573 22877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132: Neither blocks nor log segments found. Creating new log.
I20260927 21:32:34.451023 22877 log.cc:824] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132: Log is configured to *not* fsync() on all Append() calls
I20260927 21:32:34.454818 22877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132: No bootstrap required, opened a new log
I20260927 21:32:34.463416 22877 raft_consensus.cc:348] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276cf1185cf4410a917fadef3a52132" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41659 } } peers { permanent_uuid: "4e91409a5a9e4439a7fad9d75badefb8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "f60e2c1e7dd640d3a4732445af9d3a5e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } }
I20260927 21:32:34.463960 22877 raft_consensus.cc:729] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276cf1185cf4410a917fadef3a52132, State: Initialized, Role: FOLLOWER
I20260927 21:32:34.464607 22877 consensus_queue.cc:260] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276cf1185cf4410a917fadef3a52132" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41659 } } peers { permanent_uuid: "4e91409a5a9e4439a7fad9d75badefb8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "f60e2c1e7dd640d3a4732445af9d3a5e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } }
I20260927 21:32:34.466286 23046 sys_catalog.cc:455] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276cf1185cf4410a917fadef3a52132" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41659 } } peers { permanent_uuid: "4e91409a5a9e4439a7fad9d75badefb8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "f60e2c1e7dd640d3a4732445af9d3a5e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } } }
I20260927 21:32:34.467033 23046 sys_catalog.cc:458] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:32:34.467463 22877 sys_catalog.cc:564] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:32:34.467522 22852 raft_consensus.cc:482] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 21:32:34.467749 22852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:32:34.472366 22852 raft_consensus.cc:504] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276cf1185cf4410a917fadef3a52132" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41659 } } peers { permanent_uuid: "4e91409a5a9e4439a7fad9d75badefb8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "f60e2c1e7dd640d3a4732445af9d3a5e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } }
I20260927 21:32:34.475200 22852 leader_election.cc:288] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132 [CANDIDATE]: Term 1 election: Requested vote from peers 4e91409a5a9e4439a7fad9d75badefb8 (127.0.97.61:42509), f60e2c1e7dd640d3a4732445af9d3a5e (127.0.97.60:38809)
I20260927 21:32:34.475760 23000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b276cf1185cf4410a917fadef3a52132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f60e2c1e7dd640d3a4732445af9d3a5e"
I20260927 21:32:34.476296 23000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f60e2c1e7dd640d3a4732445af9d3a5e [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:32:34.480006 23000 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f60e2c1e7dd640d3a4732445af9d3a5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276cf1185cf4410a917fadef3a52132 in term 1.
W20260927 21:32:34.493168 23057 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:32:34.493394 23057 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:32:34.494328 22826 leader_election.cc:302] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276cf1185cf4410a917fadef3a52132, f60e2c1e7dd640d3a4732445af9d3a5e; no voters:
I20260927 21:32:34.495005 23046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:32:34.484869 22810 master_runner.cc:186] Error getting master registration for 127.0.97.61:42509: OK, instance_id { permanent_uuid: "4e91409a5a9e4439a7fad9d75badefb8" instance_seqno: 1790544753787297 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:32:34.498728 22810 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:32:34.499189 23046 raft_consensus.cc:686] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132 [term 1 LEADER]: Becoming Leader. State: Replica: b276cf1185cf4410a917fadef3a52132, State: Running, Role: LEADER
I20260927 21:32:34.499954 23046 consensus_queue.cc:237] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276cf1185cf4410a917fadef3a52132" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41659 } } peers { permanent_uuid: "4e91409a5a9e4439a7fad9d75badefb8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "f60e2c1e7dd640d3a4732445af9d3a5e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } }
I20260927 21:32:34.514024 23060 sys_catalog.cc:455] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b276cf1185cf4410a917fadef3a52132. Latest consensus state: current_term: 1 leader_uuid: "b276cf1185cf4410a917fadef3a52132" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276cf1185cf4410a917fadef3a52132" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41659 } } peers { permanent_uuid: "4e91409a5a9e4439a7fad9d75badefb8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "f60e2c1e7dd640d3a4732445af9d3a5e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } } }
I20260927 21:32:34.514583 23060 sys_catalog.cc:458] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:34.516131 22951 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } attempt: 5
W20260927 21:32:34.518286 22831 tablet.cc:2369] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:32:34.522950 23066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:32:34.525492 23066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:32:34.537678 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.513199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44156 (local address 127.0.97.62:41659)
0927 21:32:34.513418 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:34.513425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:34.514203 (+ 778us) server_negotiation.cc:407] Connection header received
0927 21:32:34.514527 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:34.514534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:34.514632 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:34.514725 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:34.517499 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.518719 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.521167 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.522049 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.522192 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:34.527368 (+ 5176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:34.527379 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:34.527388 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:34.527461 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:34.532778 (+ 5317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:34.534112 (+ 1334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:34.534122 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:34.534128 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:34.534323 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:34.534937 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:34.534946 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:34.534952 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:34.535512 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:34.535854 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:34.536675 (+ 821us) server_negotiation.cc:299] Negotiation successful
0927 21:32:34.537011 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 21:32:34.549326 22979 tablet.cc:2369] T 00000000000000000000000000000000 P f60e2c1e7dd640d3a4732445af9d3a5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:32:34.555725 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.516024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36060 (local address 127.0.97.60:38809)
0927 21:32:34.516240 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:34.516248 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:34.522280 (+ 6032us) server_negotiation.cc:407] Connection header received
0927 21:32:34.529957 (+ 7677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:34.529965 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:34.530076 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:34.530156 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:34.537740 (+ 7584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.538794 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.541226 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.541985 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.542100 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:34.545554 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:34.545561 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:34.545569 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:34.545638 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:34.550618 (+ 4980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:34.553367 (+ 2749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:34.553373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:34.553380 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:34.553540 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:34.553994 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:34.554000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:34.554004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:34.554345 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:34.554564 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:34.555128 (+ 564us) server_negotiation.cc:299] Negotiation successful
0927 21:32:34.555377 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 21:32:34 dist-test-slave-bxm0 krb5kdc[22797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 21:32:34.563202 23000 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f60e2c1e7dd640d3a4732445af9d3a5e [term 1 FOLLOWER]: Refusing update from remote peer b276cf1185cf4410a917fadef3a52132: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:32:34.564311 23046 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f60e2c1e7dd640d3a4732445af9d3a5e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:32:34.568629 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.519113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40374 (local address 127.0.97.61:42509)
0927 21:32:34.519292 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:34.519298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:34.539038 (+ 19740us) server_negotiation.cc:407] Connection header received
0927 21:32:34.539361 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:34.539368 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:34.539457 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:34.539535 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:34.541769 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.542867 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.546145 (+ 3278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.546849 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.546968 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:34.556448 (+ 9480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:34.556454 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:34.556461 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:34.556507 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:34.561666 (+ 5159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:34.565234 (+ 3568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:34.565241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:34.565245 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:34.565427 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:34.565999 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:34.566007 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:34.566010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:34.566428 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:34.566716 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:34.567953 (+ 1237us) server_negotiation.cc:299] Negotiation successful
0927 21:32:34.568211 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 21:32:34.579481 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.516718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38809 (local address 127.0.0.1:36062)
0927 21:32:34.516988 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:34.517007 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:32:34.517108 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:32:34.540192 (+ 23084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:32:34.540198 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:32:34.540711 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:32:34.540964 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:34.540978 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.549689 (+ 8711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.549708 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:34.551171 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:34.551188 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.552307 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.552314 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:34.552562 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:34.553265 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:32:34.553292 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:32:34.569489 (+ 16197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:32:34.575079 (+ 5590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:34.575087 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:34.575094 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:34.575777 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:34.576289 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:34.576298 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:34.576303 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:34.576510 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:34.578148 (+ 1638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:32:34.578156 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:32:34.578365 (+ 209us) client_negotiation.cc:769] Sending connection context
0927 21:32:34.578559 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 21:32:34.578921 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260927 21:32:34.580701 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.517420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36062 (local address 127.0.97.60:38809)
0927 21:32:34.539228 (+ 21808us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:34.539236 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:34.539352 (+ 116us) server_negotiation.cc:407] Connection header received
0927 21:32:34.539497 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:34.539502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:34.539865 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:34.541240 (+ 1375us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:34.541483 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.542753 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.551376 (+ 8623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.552121 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.552265 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:34.569745 (+ 17480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:34.569752 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:34.569758 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:34.569800 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:34.574865 (+ 5065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:34.575954 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:34.575962 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:34.575969 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:34.576140 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:34.576672 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:34.576680 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:34.576686 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:34.577873 (+ 1187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:34.578060 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:34.578860 (+ 800us) server_negotiation.cc:299] Negotiation successful
0927 21:32:34.579446 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21258,"thread_start_us":20609,"threads_started":1}
I20260927 21:32:34.587723 22951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e91409a5a9e4439a7fad9d75badefb8: Bootstrap starting.
I20260927 21:32:34.590004 23032 sys_catalog.cc:455] T 00000000000000000000000000000000 P f60e2c1e7dd640d3a4732445af9d3a5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b276cf1185cf4410a917fadef3a52132. Latest consensus state: current_term: 1 leader_uuid: "b276cf1185cf4410a917fadef3a52132" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276cf1185cf4410a917fadef3a52132" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41659 } } peers { permanent_uuid: "4e91409a5a9e4439a7fad9d75badefb8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "f60e2c1e7dd640d3a4732445af9d3a5e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } } }
I20260927 21:32:34.590314 23032 sys_catalog.cc:458] T 00000000000000000000000000000000 P f60e2c1e7dd640d3a4732445af9d3a5e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:32:34.594278 23070 mvcc.cc:205] Tried to move back new op lower bound from 7334071314676830208 to 7334071314439991296. Current Snapshot: MvccSnapshot[applied={T|T < 7334071314676830208}]
I20260927 21:32:34.596014 22951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e91409a5a9e4439a7fad9d75badefb8: Neither blocks nor log segments found. Creating new log.
I20260927 21:32:34.600389 22951 log.cc:824] T 00000000000000000000000000000000 P 4e91409a5a9e4439a7fad9d75badefb8: Log is configured to *not* fsync() on all Append() calls
I20260927 21:32:34.619081 22951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e91409a5a9e4439a7fad9d75badefb8: No bootstrap required, opened a new log
I20260927 21:32:34.621060 23060 sys_catalog.cc:455] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b276cf1185cf4410a917fadef3a52132" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276cf1185cf4410a917fadef3a52132" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41659 } } peers { permanent_uuid: "4e91409a5a9e4439a7fad9d75badefb8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "f60e2c1e7dd640d3a4732445af9d3a5e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } } }
I20260927 21:32:34.621387 23060 sys_catalog.cc:458] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:34.625981 23066 catalog_manager.cc:1347] Generated new cluster ID: 970f99ac441e44fabaa4e01df7faab77
I20260927 21:32:34.626114 23066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:32:34.627173 23032 sys_catalog.cc:455] T 00000000000000000000000000000000 P f60e2c1e7dd640d3a4732445af9d3a5e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b276cf1185cf4410a917fadef3a52132" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276cf1185cf4410a917fadef3a52132" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41659 } } peers { permanent_uuid: "4e91409a5a9e4439a7fad9d75badefb8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "f60e2c1e7dd640d3a4732445af9d3a5e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } } }
I20260927 21:32:34.627445 23032 sys_catalog.cc:458] T 00000000000000000000000000000000 P f60e2c1e7dd640d3a4732445af9d3a5e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:32:34.644767 22951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e91409a5a9e4439a7fad9d75badefb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276cf1185cf4410a917fadef3a52132" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41659 } } peers { permanent_uuid: "4e91409a5a9e4439a7fad9d75badefb8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "f60e2c1e7dd640d3a4732445af9d3a5e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } }
I20260927 21:32:34.645233 22951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e91409a5a9e4439a7fad9d75badefb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e91409a5a9e4439a7fad9d75badefb8, State: Initialized, Role: FOLLOWER
I20260927 21:32:34.645838 22951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e91409a5a9e4439a7fad9d75badefb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276cf1185cf4410a917fadef3a52132" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41659 } } peers { permanent_uuid: "4e91409a5a9e4439a7fad9d75badefb8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "f60e2c1e7dd640d3a4732445af9d3a5e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } }
I20260927 21:32:34.655951 23066 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:32:34.657943 23066 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:32:34.659289 23076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e91409a5a9e4439a7fad9d75badefb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276cf1185cf4410a917fadef3a52132" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41659 } } peers { permanent_uuid: "4e91409a5a9e4439a7fad9d75badefb8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "f60e2c1e7dd640d3a4732445af9d3a5e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } } }
I20260927 21:32:34.660580 22951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e91409a5a9e4439a7fad9d75badefb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:32:34.660663 22926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b276cf1185cf4410a917fadef3a52132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4e91409a5a9e4439a7fad9d75badefb8"
I20260927 21:32:34.661082 22925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e91409a5a9e4439a7fad9d75badefb8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:32:34.665042 22925 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4e91409a5a9e4439a7fad9d75badefb8 [term 1 FOLLOWER]: Refusing update from remote peer b276cf1185cf4410a917fadef3a52132: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:32:34.668468 22926 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4e91409a5a9e4439a7fad9d75badefb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b276cf1185cf4410a917fadef3a52132 in term 1.
I20260927 21:32:34.670661 23076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e91409a5a9e4439a7fad9d75badefb8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:32:34.670670 23046 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e91409a5a9e4439a7fad9d75badefb8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:32:34.690187 23066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132: Generated new TSK 0
I20260927 21:32:34.691288 23066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:32:34.735118 23076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e91409a5a9e4439a7fad9d75badefb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b276cf1185cf4410a917fadef3a52132. Latest consensus state: current_term: 1 leader_uuid: "b276cf1185cf4410a917fadef3a52132" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276cf1185cf4410a917fadef3a52132" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41659 } } peers { permanent_uuid: "4e91409a5a9e4439a7fad9d75badefb8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "f60e2c1e7dd640d3a4732445af9d3a5e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } } }
I20260927 21:32:34.735450 23076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e91409a5a9e4439a7fad9d75badefb8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:32:34.735962 23060 sys_catalog.cc:455] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b276cf1185cf4410a917fadef3a52132" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276cf1185cf4410a917fadef3a52132" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41659 } } peers { permanent_uuid: "4e91409a5a9e4439a7fad9d75badefb8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "f60e2c1e7dd640d3a4732445af9d3a5e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } } }
I20260927 21:32:34.736248 23060 sys_catalog.cc:458] T 00000000000000000000000000000000 P b276cf1185cf4410a917fadef3a52132 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:34.736605 23078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e91409a5a9e4439a7fad9d75badefb8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b276cf1185cf4410a917fadef3a52132" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b276cf1185cf4410a917fadef3a52132" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41659 } } peers { permanent_uuid: "4e91409a5a9e4439a7fad9d75badefb8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "f60e2c1e7dd640d3a4732445af9d3a5e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38809 } } }
I20260927 21:32:34.736970 23078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e91409a5a9e4439a7fad9d75badefb8 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 21:32:34.824687 23029 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:32:34.808969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44160 (local address 127.0.97.62:41659)
0927 21:32:34.809502 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:34.809512 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:34.822223 (+ 12711us) server_negotiation.cc:407] Connection header received
0927 21:32:34.822573 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:34.822579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:34.822675 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:34.822760 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:34.824329 (+ 1569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44160: BlockingRecv error: recv got EOF from 127.0.0.1:44160 (error 108)
Metrics: {"server-negotiator.queue_time_us":306}
W20260927 21:32:34.827919 22955 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:32:34.813432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40390 (local address 127.0.97.61:42509)
0927 21:32:34.813720 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:34.813730 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:34.825689 (+ 11959us) server_negotiation.cc:407] Connection header received
0927 21:32:34.826010 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:34.826020 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:34.826138 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:34.826224 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:34.827646 (+ 1422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40390: BlockingRecv error: recv got EOF from 127.0.0.1:40390 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260927 21:32:34.832340 23068 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:32:34.822411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36078 (local address 127.0.97.60:38809)
0927 21:32:34.822620 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:34.822632 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:34.827129 (+ 4497us) server_negotiation.cc:407] Connection header received
0927 21:32:34.830594 (+ 3465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:34.830601 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:34.830707 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:34.830788 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:34.832086 (+ 1298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36078: BlockingRecv error: recv got EOF from 127.0.0.1:36078 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 21:32:34.838959 23096 catalog_manager.cc:1259] Loaded cluster ID: 970f99ac441e44fabaa4e01df7faab77
I20260927 21:32:34.839074 23096 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4e91409a5a9e4439a7fad9d75badefb8: loading cluster ID for follower catalog manager: success
I20260927 21:32:34.852823 23096 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4e91409a5a9e4439a7fad9d75badefb8: acquiring CA information for follower catalog manager: success
I20260927 21:32:34.856215 23096 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4e91409a5a9e4439a7fad9d75badefb8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:32:34.871109 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.851601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40394 (local address 127.0.97.61:42509)
0927 21:32:34.851764 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:34.851769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:34.863073 (+ 11304us) server_negotiation.cc:407] Connection header received
0927 21:32:34.863660 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:34.863665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:34.863855 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:34.863982 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:32:34.864703 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.865906 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.868032 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.868707 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.868805 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:34.869545 (+ 740us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:32:34.870274 (+ 729us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:32:34.870388 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:34.870644 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 21:32:34.870743 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 21:32:34.893826 23068 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:32:34.878952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36088 (local address 127.0.97.60:38809)
0927 21:32:34.879129 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:34.879135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:34.891505 (+ 12370us) server_negotiation.cc:407] Connection header received
0927 21:32:34.892117 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:34.892123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:34.892255 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:34.892340 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:34.893504 (+ 1164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36088: BlockingRecv error: recv got EOF from 127.0.0.1:36088 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 21:32:34.947192 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:34.931781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40404 (local address 127.0.97.61:42509)
0927 21:32:34.931977 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:34.931986 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:34.939916 (+ 7930us) server_negotiation.cc:407] Connection header received
0927 21:32:34.940460 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:34.940467 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:34.940568 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:34.940651 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:32:34.941374 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.942667 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.944751 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:34.945403 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:34.945500 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:34.946175 (+ 675us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:32:34.946343 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:32:34.946432 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:34.946739 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 21:32:34.946840 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 21:32:34.992000 23068 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:32:34.981664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36094 (local address 127.0.97.60:38809)
0927 21:32:34.981885 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:34.981894 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:34.990261 (+ 8367us) server_negotiation.cc:407] Connection header received
0927 21:32:34.990501 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:34.990506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:34.990619 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:34.990723 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:34.991761 (+ 1038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36094: BlockingRecv error: recv got EOF from 127.0.0.1:36094 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 21:32:35.039479 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:35.024811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40420 (local address 127.0.97.61:42509)
0927 21:32:35.025026 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:35.025033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:35.032363 (+ 7330us) server_negotiation.cc:407] Connection header received
0927 21:32:35.032855 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:35.032861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:35.032951 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:35.033021 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:32:35.033753 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:35.034946 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:35.037058 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:35.037814 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:35.037924 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:35.038558 (+ 634us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:32:35.038715 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:32:35.038784 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:35.039020 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 21:32:35.039109 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260927 21:32:35.066781 23068 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:32:35.062130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36098 (local address 127.0.97.60:38809)
0927 21:32:35.062318 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:35.062325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:35.065043 (+ 2718us) server_negotiation.cc:407] Connection header received
0927 21:32:35.065217 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:35.065225 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:35.065334 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:35.065475 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:35.066498 (+ 1023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36098: BlockingRecv error: recv got EOF from 127.0.0.1:36098 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 21:32:35.082505 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:35.075069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40428 (local address 127.0.97.61:42509)
0927 21:32:35.075302 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:35.075309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:35.075898 (+ 589us) server_negotiation.cc:407] Connection header received
0927 21:32:35.076100 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:35.076109 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:35.076216 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:35.076338 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:32:35.076861 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:35.078114 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:35.080105 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:35.080948 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:35.081129 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:35.081581 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:32:35.081770 (+ 189us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:32:35.081883 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:35.082063 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 21:32:35.082163 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
W20260927 21:32:35.094303 23068 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:32:35.091739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36106 (local address 127.0.97.60:38809)
0927 21:32:35.092018 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:35.092024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:35.092572 (+ 548us) server_negotiation.cc:407] Connection header received
0927 21:32:35.092784 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:35.092790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:35.092901 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:35.093112 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:35.093978 (+ 866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36106: BlockingRecv error: recv got EOF from 127.0.0.1:36106 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
I20260927 21:32:35.113214 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:35.105824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40432 (local address 127.0.97.61:42509)
0927 21:32:35.106084 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:35.106093 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:35.106710 (+ 617us) server_negotiation.cc:407] Connection header received
0927 21:32:35.106937 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:35.106943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:35.107035 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:35.107138 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:32:35.107646 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:35.108800 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:35.110849 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:35.111572 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:35.111709 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:35.112310 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:32:35.112484 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:32:35.112586 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:35.112783 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 21:32:35.112871 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
W20260927 21:32:35.129758 23068 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:32:35.127235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36120 (local address 127.0.97.60:38809)
0927 21:32:35.127415 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:35.127422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:35.127973 (+ 551us) server_negotiation.cc:407] Connection header received
0927 21:32:35.128292 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:35.128300 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:35.128411 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:35.128545 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:35.129442 (+ 897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36120: BlockingRecv error: recv got EOF from 127.0.0.1:36120 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:32:35.159240 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:35.151696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40442 (local address 127.0.97.61:42509)
0927 21:32:35.151882 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:35.151888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:35.152442 (+ 554us) server_negotiation.cc:407] Connection header received
0927 21:32:35.152675 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:35.152683 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:35.152788 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:35.152912 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:32:35.153421 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:35.154729 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:35.156902 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:35.157647 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:35.157803 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:35.158303 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:32:35.158461 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:32:35.158574 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:35.158749 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 21:32:35.158857 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 21:32:35.174156 23068 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:32:35.171598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36134 (local address 127.0.97.60:38809)
0927 21:32:35.171803 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:35.171810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:35.172405 (+ 595us) server_negotiation.cc:407] Connection header received
0927 21:32:35.172600 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:35.172607 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:35.172709 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:35.172823 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:35.173890 (+ 1067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36134: BlockingRecv error: recv got EOF from 127.0.0.1:36134 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 21:32:35.216575 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:35.209894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40456 (local address 127.0.97.61:42509)
0927 21:32:35.210181 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:35.210187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:35.210488 (+ 301us) server_negotiation.cc:407] Connection header received
0927 21:32:35.210729 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:35.210733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:35.210825 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:35.210903 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:32:35.211450 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:35.212383 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:35.214505 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:35.215018 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:35.215142 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:35.215686 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:32:35.215801 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:32:35.215883 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:35.216082 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 21:32:35.216158 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260927 21:32:35.225158 23068 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:32:35.223037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36148 (local address 127.0.97.60:38809)
0927 21:32:35.223256 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:35.223264 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:35.223701 (+ 437us) server_negotiation.cc:407] Connection header received
0927 21:32:35.223905 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:35.223911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:35.224019 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:35.224139 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:35.224900 (+ 761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36148: BlockingRecv error: recv got EOF from 127.0.0.1:36148 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 21:32:35.311687 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:35.304737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40462 (local address 127.0.97.61:42509)
0927 21:32:35.304977 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:35.304984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:35.305581 (+ 597us) server_negotiation.cc:407] Connection header received
0927 21:32:35.305856 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:35.305862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:35.305950 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:35.306056 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:32:35.306529 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:35.307731 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:35.309772 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:35.310307 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:35.310413 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:35.310953 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:32:35.311062 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:32:35.311140 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:35.311312 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 21:32:35.311377 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
W20260927 21:32:35.326308 23068 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:32:35.323938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36160 (local address 127.0.97.60:38809)
0927 21:32:35.324121 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:35.324127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:35.324630 (+ 503us) server_negotiation.cc:407] Connection header received
0927 21:32:35.324844 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:35.324850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:35.324942 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:35.325039 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:35.326095 (+ 1056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36160: BlockingRecv error: recv got EOF from 127.0.0.1:36160 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:32:35.414363 23043 catalog_manager.cc:1259] Loaded cluster ID: 970f99ac441e44fabaa4e01df7faab77
I20260927 21:32:35.414557 23043 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f60e2c1e7dd640d3a4732445af9d3a5e: loading cluster ID for follower catalog manager: success
I20260927 21:32:35.417016 23043 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f60e2c1e7dd640d3a4732445af9d3a5e: acquiring CA information for follower catalog manager: success
I20260927 21:32:35.418916 23043 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f60e2c1e7dd640d3a4732445af9d3a5e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:32:35.463037 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:35.457826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40470 (local address 127.0.97.61:42509)
0927 21:32:35.458035 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:35.458040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:35.458407 (+ 367us) server_negotiation.cc:407] Connection header received
0927 21:32:35.458568 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:35.458572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:35.458639 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:35.458741 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:32:35.459102 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:35.459795 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:35.461342 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:35.461835 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:35.461946 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:35.462393 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:32:35.462542 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:32:35.462614 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:35.462745 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 21:32:35.462817 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 21:32:35.479736 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:35.473752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36162 (local address 127.0.97.60:38809)
0927 21:32:35.473989 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:35.473997 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:35.474305 (+ 308us) server_negotiation.cc:407] Connection header received
0927 21:32:35.474525 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:35.474531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:35.474809 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:35.474935 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:32:35.475381 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:35.476095 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:35.477431 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:35.477923 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:35.478063 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:35.478585 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:32:35.479115 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:32:35.479286 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:35.479343 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 21:32:35.479429 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 21:32:35.493742 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:35.485982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40482 (local address 127.0.97.61:42509)
0927 21:32:35.486230 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:35.486240 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:35.487482 (+ 1242us) server_negotiation.cc:407] Connection header received
0927 21:32:35.487682 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:35.487690 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:35.487800 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:35.487894 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:32:35.488728 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:35.489877 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:35.491820 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:35.492311 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:35.492433 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:35.492921 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:32:35.493048 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:32:35.493141 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:35.493307 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 21:32:35.493399 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 21:32:35.494212 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:35.485855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44162 (local address 127.0.97.62:41659)
0927 21:32:35.486704 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:35.486712 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:35.487262 (+ 550us) server_negotiation.cc:407] Connection header received
0927 21:32:35.487436 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:35.487445 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:35.487725 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:35.487907 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:32:35.488557 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:35.489290 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:35.491392 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:35.492060 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:35.492201 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:35.492731 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:32:35.493208 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:32:35.493329 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:35.493827 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 21:32:35.493919 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":523,"threads_started":1}
I20260927 21:32:35.494517 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:35.487224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36178 (local address 127.0.97.60:38809)
0927 21:32:35.487405 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:35.487413 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:35.488206 (+ 793us) server_negotiation.cc:407] Connection header received
0927 21:32:35.488491 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:35.488497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:35.488583 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:35.488646 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:32:35.489439 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:35.490338 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:35.492201 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:35.492821 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:35.492906 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:35.493457 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:32:35.493639 (+ 182us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:32:35.493712 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:35.494138 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 21:32:35.494226 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 21:32:35.499313 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 22810
I20260927 21:32:35.500428 22872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:32:35.610586 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 22810
I20260927 21:32:35.633899 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 22884
I20260927 21:32:35.634979 22946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:32:35.742816 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 22884
I20260927 21:32:35.762960 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 22958
I20260927 21:32:35.764060 23020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:32:35.871176 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 22958
2026-09-27T21:32:35Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4393 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:32:35 dist-test-slave-bxm0 krb5kdc[23207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:32:35 dist-test-slave-bxm0 krb5kdc[23207](info): set up 2 sockets
Sep 27 21:32:35 dist-test-slave-bxm0 krb5kdc[23207](info): commencing operation
krb5kdc: starting...
W20260927 21:32:36.978212 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/krb5kdc/test-user.keytab.
Sep 27 21:32:37 dist-test-slave-bxm0 krb5kdc[23207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:32:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:32:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 21:32:37.241472 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34953
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34953,127.0.97.61:44775,127.0.97.60:36099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:32:37.574309 23220 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:32:37.574671 23220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:37.574802 23220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:37.574937 23220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:37.582321 23220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:32:37.582414 23220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:37.582516 23220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:37.582615 23220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:32:37.582732 23220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:32:37.593395 23220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34953,127.0.97.61:44775,127.0.97.60:36099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:37.594729 23220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:37.596066 23220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:37.607936 23228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:37.608012 23226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:37.608008 23225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:37.609174 23220 server_base.cc:1034] running on GCE node
I20260927 21:32:37.610054 23220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:37.611457 23220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:37.612712 23220 hybrid_clock.cc:648] HybridClock initialized: now 1790544757612635 us; error 44 us; skew 500 ppm
I20260927 21:32:37.613041 23220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:37 dist-test-slave-bxm0 krb5kdc[23207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:37.616155 23220 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 21:32:37.617690 23220 webserver.cc:460] Webserver started at http://127.0.97.62:43903/ using document root <none> and password file <none>
I20260927 21:32:37.618381 23220 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:37.618512 23220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:37.618834 23220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:37.622570 23220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-0/data/instance:
uuid: "191dc9d656e3456cba924abf9b41c441"
format_stamp: "Formatted at 2026-09-27 21:32:37 on dist-test-slave-bxm0"
server_key: "a55c62a40a8bfa3c740e10d08e33024b"
server_key_iv: "cbd52376e9d7e56f3b33b1b9ecfb30af"
server_key_version: "encryptionkey@0"
I20260927 21:32:37.623857 23220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-0/wal/instance:
uuid: "191dc9d656e3456cba924abf9b41c441"
format_stamp: "Formatted at 2026-09-27 21:32:37 on dist-test-slave-bxm0"
server_key: "a55c62a40a8bfa3c740e10d08e33024b"
server_key_iv: "cbd52376e9d7e56f3b33b1b9ecfb30af"
server_key_version: "encryptionkey@0"
I20260927 21:32:37.631201 23220 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.006s
I20260927 21:32:37.634773 23235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:37.636078 23220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 21:32:37.636266 23220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-0/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-0/wal
uuid: "191dc9d656e3456cba924abf9b41c441"
format_stamp: "Formatted at 2026-09-27 21:32:37 on dist-test-slave-bxm0"
server_key: "a55c62a40a8bfa3c740e10d08e33024b"
server_key_iv: "cbd52376e9d7e56f3b33b1b9ecfb30af"
server_key_version: "encryptionkey@0"
I20260927 21:32:37.636492 23220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:37.652233 23220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:37.656288 23220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:37.656575 23220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:37.675489 23220 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34953
I20260927 21:32:37.675514 23286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34953 every 8 connection(s)
I20260927 21:32:37.676862 23220 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-0/data/info.pb
I20260927 21:32:37.677263 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 23220
I20260927 21:32:37.677456 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-0/wal/instance
I20260927 21:32:37.678115 388 external_mini_cluster.cc:1428] Setting key 8f76488e20a1d0165e243afaa4192861
I20260927 21:32:37.680866 23287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:32:37.687872 23287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34953 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 27 21:32:37 dist-test-slave-bxm0 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:32:37.715890 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:37.691361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36288 (local address 127.0.97.62:34953)
0927 21:32:37.693552 (+ 2191us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:37.693564 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:37.693681 (+ 117us) server_negotiation.cc:407] Connection header received
0927 21:32:37.695858 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:37.695864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:37.696114 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:37.696279 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:37.698110 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:37.699334 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:37.701780 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:37.702341 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:37.702473 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:37.706979 (+ 4506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:37.706993 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:37.707015 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:37.707063 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:37.710890 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:37.711774 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:37.711786 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:37.711792 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:37.711998 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:37.712458 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:37.712466 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:37.712472 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:37.713233 (+ 761us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:37.713450 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:37.714138 (+ 688us) server_negotiation.cc:299] Negotiation successful
0927 21:32:37.714706 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":207,"server-negotiator.queue_time_us":1864,"spinlock_wait_cycles":17152,"thread_start_us":1610,"threads_started":1}
I20260927 21:32:37.715958 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:37.691361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34953 (local address 127.0.0.1:36288)
0927 21:32:37.692390 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:37.692454 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 21:32:37.695455 (+ 3001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:32:37.696385 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:32:37.696398 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:32:37.697295 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:32:37.697875 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:37.697909 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:37.699506 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:37.699515 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:37.701591 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:37.701648 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:37.702613 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:37.702620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:37.702812 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:37.703512 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:32:37.703536 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:32:37.706687 (+ 3151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:32:37.711051 (+ 4364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:37.711065 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:37.711080 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:37.711607 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:37.712118 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:37.712125 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:37.712130 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:37.712329 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:37.713373 (+ 1044us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:32:37.713387 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:32:37.713682 (+ 295us) client_negotiation.cc:769] Sending connection context
0927 21:32:37.713942 (+ 260us) client_negotiation.cc:240] Negotiation successful
0927 21:32:37.714487 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":446,"threads_started":1}
I20260927 21:32:37.719111 23287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44775 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:32:37.720319 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:37.719878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44775 (local address 127.0.0.1:40172)
0927 21:32:37.720070 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:37.720198 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 21:32:37.720651 23236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44775: connect: Connection refused (error 111)
I20260927 21:32:37.722251 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:37.721914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44775 (local address 127.0.0.1:40182)
0927 21:32:37.722049 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:37.722118 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 21:32:37.722575 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44775: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44775: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260927 21:32:37.740913 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44775
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34953,127.0.97.61:44775,127.0.97.60:36099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:32:37.771965 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44775 } attempt: 1
I20260927 21:32:37.773202 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:37.772758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44775 (local address 127.0.0.1:40198)
0927 21:32:37.772957 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:37.773049 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 21:32:37.774281 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:37.773915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44775 (local address 127.0.0.1:40208)
0927 21:32:37.774066 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:37.774144 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 21:32:37.774464 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44775: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44775: connect: Connection refused (error 111)
I20260927 21:32:37.842726 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44775 } attempt: 2
I20260927 21:32:37.843911 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:37.843429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44775 (local address 127.0.0.1:40216)
0927 21:32:37.843689 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:37.843778 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 21:32:37.844904 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:37.844526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44775 (local address 127.0.0.1:40226)
0927 21:32:37.844683 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:37.844758 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 21:32:37.845113 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44775: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44775: connect: Connection refused (error 111)
I20260927 21:32:37.936345 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44775 } attempt: 3
I20260927 21:32:37.937584 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:37.937150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44775 (local address 127.0.0.1:40228)
0927 21:32:37.937356 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:37.937453 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 21:32:37.938635 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:37.938292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44775 (local address 127.0.0.1:40232)
0927 21:32:37.938431 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:37.938507 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 21:32:37.938814 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44775: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44775: connect: Connection refused (error 111)
W20260927 21:32:38.072379 23294 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:32:38.072774 23294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:38.072901 23294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:38.073035 23294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:32:38.080664 23294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:32:38.080848 23294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:38.080973 23294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:38.081063 23294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:32:38.081171 23294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:32:38.082080 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44775 } attempt: 4
I20260927 21:32:38.083395 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.082767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44775 (local address 127.0.0.1:40248)
0927 21:32:38.083093 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.083227 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260927 21:32:38.084549 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.084137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44775 (local address 127.0.0.1:40264)
0927 21:32:38.084299 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.084396 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 21:32:38.084757 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44775: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44775: connect: Connection refused (error 111)
I20260927 21:32:38.092367 23294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34953,127.0.97.61:44775,127.0.97.60:36099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:38.093753 23294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:38.095086 23294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:38.106843 23299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:38.107036 23300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:38.107201 23302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:38.107733 23294 server_base.cc:1034] running on GCE node
I20260927 21:32:38.108605 23294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:38.110070 23294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:38.111320 23294 hybrid_clock.cc:648] HybridClock initialized: now 1790544758111239 us; error 52 us; skew 500 ppm
I20260927 21:32:38.111649 23294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 21:32:38 dist-test-slave-bxm0 krb5kdc[23207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:38.114897 23294 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260927 21:32:38.116396 23294 webserver.cc:460] Webserver started at http://127.0.97.61:45403/ using document root <none> and password file <none>
I20260927 21:32:38.117125 23294 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:38.117252 23294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:38.117597 23294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:38.120707 23294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-1/data/instance:
uuid: "e69011d81fe843248c42fc696460fd64"
format_stamp: "Formatted at 2026-09-27 21:32:38 on dist-test-slave-bxm0"
server_key: "67acca4075b109715306a827da26b330"
server_key_iv: "7ed2bbf73d684076260eb4abfae72159"
server_key_version: "encryptionkey@0"
I20260927 21:32:38.121575 23294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-1/wal/instance:
uuid: "e69011d81fe843248c42fc696460fd64"
format_stamp: "Formatted at 2026-09-27 21:32:38 on dist-test-slave-bxm0"
server_key: "67acca4075b109715306a827da26b330"
server_key_iv: "7ed2bbf73d684076260eb4abfae72159"
server_key_version: "encryptionkey@0"
I20260927 21:32:38.126897 23294 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 21:32:38.130426 23309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:38.131712 23294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 21:32:38.131902 23294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-1/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-1/wal
uuid: "e69011d81fe843248c42fc696460fd64"
format_stamp: "Formatted at 2026-09-27 21:32:38 on dist-test-slave-bxm0"
server_key: "67acca4075b109715306a827da26b330"
server_key_iv: "7ed2bbf73d684076260eb4abfae72159"
server_key_version: "encryptionkey@0"
I20260927 21:32:38.132118 23294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:38.152525 23294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:38.156817 23294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:38.157120 23294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:38.175500 23294 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:44775
I20260927 21:32:38.175520 23360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:44775 every 8 connection(s)
I20260927 21:32:38.176882 23294 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-1/data/info.pb
I20260927 21:32:38.180590 23361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:32:38.186579 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 23294
I20260927 21:32:38.186781 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-1/wal/instance
I20260927 21:32:38.187387 388 external_mini_cluster.cc:1428] Setting key 4d86e06a5f9b235b792c820df00c991a
I20260927 21:32:38.187398 23361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34953 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:32:38 dist-test-slave-bxm0 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:32:38.210479 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.190716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36290 (local address 127.0.97.62:34953)
0927 21:32:38.190961 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:38.190970 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:38.192181 (+ 1211us) server_negotiation.cc:407] Connection header received
0927 21:32:38.194193 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:38.194201 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:38.194297 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:38.194423 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:38.196295 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.196973 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.198584 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.199007 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.199146 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:38.202937 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:38.202946 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:38.202954 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:38.203003 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:38.206860 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.207654 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.207663 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.207670 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.207845 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.208279 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.208287 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.208292 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.208729 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:38.208908 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:38.209752 (+ 844us) server_negotiation.cc:299] Negotiation successful
0927 21:32:38.210039 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 21:32:38.211148 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.191137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34953 (local address 127.0.0.1:36290)
0927 21:32:38.191989 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.192044 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 21:32:38.193922 (+ 1878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:32:38.194836 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:32:38.194850 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:32:38.195552 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:32:38.196105 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:38.196121 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.197138 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.197149 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:38.198430 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:38.198442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.199234 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.199241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:38.199469 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:38.200173 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:32:38.200197 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:32:38.202618 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:32:38.207026 (+ 4408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:38.207034 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:38.207048 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:38.207503 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:38.207958 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:38.207961 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:38.207972 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:38.208152 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:38.208894 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:32:38.208909 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:32:38.209292 (+ 383us) client_negotiation.cc:769] Sending connection context
0927 21:32:38.209520 (+ 228us) client_negotiation.cc:240] Negotiation successful
0927 21:32:38.209989 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":381,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260927 21:32:38.212987 23361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44775 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 27 21:32:38 dist-test-slave-bxm0 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 21:32:38.236009 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.213881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44775 (local address 127.0.0.1:40270)
0927 21:32:38.214103 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.214125 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 21:32:38.214224 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:32:38.215196 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:32:38.215200 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:32:38.215643 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:32:38.215836 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:38.215851 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.217269 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.217275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:38.218192 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:38.218203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.219015 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.219021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:38.219209 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:38.219794 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:32:38.219811 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:32:38.222793 (+ 2982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:32:38.232950 (+ 10157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:38.232955 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:38.232958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:38.233358 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:38.233868 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:38.233872 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:38.233874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:38.234019 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:38.234831 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:32:38.234836 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:32:38.235004 (+ 168us) client_negotiation.cc:769] Sending connection context
0927 21:32:38.235189 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 21:32:38.235563 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 21:32:38.236398 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.214162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40270 (local address 127.0.97.61:44775)
0927 21:32:38.214789 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:38.214795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:38.214818 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:32:38.214860 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:38.214866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:38.215037 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:38.215158 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:38.216008 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.217109 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.218324 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.218863 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.218994 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:38.223023 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:38.223034 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:38.223049 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:38.223078 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:38.232755 (+ 9677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.233537 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.233549 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.233556 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.233752 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.234138 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.234146 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.234152 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.234671 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:38.234877 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:38.235375 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 21:32:38.235765 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":488,"thread_start_us":346,"threads_started":1}
I20260927 21:32:38.239053 23361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:32:38.240317 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.239781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36099 (local address 127.0.0.1:39734)
0927 21:32:38.239955 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.240076 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260927 21:32:38.240572 23313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
I20260927 21:32:38.242295 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.241934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36099 (local address 127.0.0.1:39738)
0927 21:32:38.242108 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.242197 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 21:32:38.242508 23361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36099: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260927 21:32:38.249153 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
/tmp/dist-test-tasklB5kC7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36099
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34953,127.0.97.61:44775,127.0.97.60:36099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:32:38.291832 23361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } attempt: 1
I20260927 21:32:38.292994 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.292540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36099 (local address 127.0.0.1:39752)
0927 21:32:38.292742 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.292836 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 21:32:38.294103 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.293772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36099 (local address 127.0.0.1:39764)
0927 21:32:38.293923 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.293998 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 21:32:38.294380 23361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36099: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
I20260927 21:32:38.362627 23361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } attempt: 2
I20260927 21:32:38.363924 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.363390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36099 (local address 127.0.0.1:39766)
0927 21:32:38.363650 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.363763 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260927 21:32:38.364959 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.364647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36099 (local address 127.0.0.1:39774)
0927 21:32:38.364773 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.364844 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 21:32:38.365197 23361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36099: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
I20260927 21:32:38.384025 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44775 } attempt: 5
Sep 27 21:32:38 dist-test-slave-bxm0 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 21:32:38.398186 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.384938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40280 (local address 127.0.97.61:44775)
0927 21:32:38.385124 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:38.385132 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:38.385153 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:32:38.385386 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:38.385390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:38.385454 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:38.385564 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:38.386445 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.387139 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.388425 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.388882 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.389021 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:38.392719 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:38.392724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:38.392727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:38.392759 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:38.395423 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.396256 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.396260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.396262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.396380 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.396797 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.396801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.396802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.397080 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:38.397191 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:38.397752 (+ 561us) server_negotiation.cc:299] Negotiation successful
0927 21:32:38.397923 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 21:32:38.398341 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.384822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44775 (local address 127.0.0.1:40280)
0927 21:32:38.385041 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.385065 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 21:32:38.385220 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:32:38.385604 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:32:38.385648 (+ 44us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:32:38.386089 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:32:38.386278 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:38.386288 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.387307 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.387332 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:38.388270 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:38.388282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.389050 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.389056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:38.389250 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:38.390014 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:32:38.390032 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:32:38.392500 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:32:38.395588 (+ 3088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:38.395595 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:38.395599 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:38.396127 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:38.396491 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:38.396496 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:38.396500 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:38.396693 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:38.397213 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:32:38.397220 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:32:38.397407 (+ 187us) client_negotiation.cc:769] Sending connection context
0927 21:32:38.397586 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 21:32:38.397925 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 21:32:38.399721 23287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:32:38.401021 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.400449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36099 (local address 127.0.0.1:39780)
0927 21:32:38.400680 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.400822 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 21:32:38.402014 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.401606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36099 (local address 127.0.0.1:39786)
0927 21:32:38.401799 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.401880 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 21:32:38.402221 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36099: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
I20260927 21:32:38.453508 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } attempt: 1
I20260927 21:32:38.455042 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.454367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36099 (local address 127.0.0.1:39798)
0927 21:32:38.454653 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.454792 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 21:32:38.456198 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.455808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36099 (local address 127.0.0.1:39814)
0927 21:32:38.455976 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.456070 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 21:32:38.456475 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36099: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
I20260927 21:32:38.456419 23361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } attempt: 3
I20260927 21:32:38.457404 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.456968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36099 (local address 127.0.0.1:39822)
0927 21:32:38.457187 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.457261 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 21:32:38.458441 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.458028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36099 (local address 127.0.0.1:39830)
0927 21:32:38.458256 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.458328 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260927 21:32:38.458638 23361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36099: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
I20260927 21:32:38.524732 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } attempt: 2
I20260927 21:32:38.526057 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.525488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36099 (local address 127.0.0.1:39836)
0927 21:32:38.525758 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.525869 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 21:32:38.527155 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.526776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36099 (local address 127.0.0.1:39846)
0927 21:32:38.526943 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.527012 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 21:32:38.527365 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36099: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
W20260927 21:32:38.594424 23368 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:32:38.594847 23368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:32:38.594978 23368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:32:38.595131 23368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 21:32:38.601858 23361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } attempt: 4
W20260927 21:32:38.602679 23368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:32:38.602813 23368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:32:38.602924 23368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:32:38.603009 23368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:32:38.603123 23368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:32:38.603154 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.602525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36099 (local address 127.0.0.1:39852)
0927 21:32:38.602768 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.602911 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 21:32:38.604182 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.603845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36099 (local address 127.0.0.1:39862)
0927 21:32:38.604000 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.604077 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 21:32:38.604417 23361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36099: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
I20260927 21:32:38.613830 23368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34953,127.0.97.61:44775,127.0.97.60:36099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 19:17:41 UTC on ccycloud.quasar-pvdqxn.root.comops.site
build id 4676136
ASAN enabled
I20260927 21:32:38.615203 23368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:32:38.616528 23368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:32:38.628502 23376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:38.628517 23374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:32:38.628545 23373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:32:38.629354 23368 server_base.cc:1034] running on GCE node
I20260927 21:32:38.630260 23368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:32:38.631705 23368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:32:38.632972 23368 hybrid_clock.cc:648] HybridClock initialized: now 1790544758632905 us; error 43 us; skew 500 ppm
I20260927 21:32:38.633354 23368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 21:32:38.633548 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } attempt: 3
I20260927 21:32:38.634618 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.634148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36099 (local address 127.0.0.1:39870)
0927 21:32:38.634376 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.634466 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 21:32:38.635736 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.635329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36099 (local address 127.0.0.1:39880)
0927 21:32:38.635545 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.635625 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260927 21:32:38.635926 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36099: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36099: connect: Connection refused (error 111)
Sep 27 21:32:38 dist-test-slave-bxm0 krb5kdc[23207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:32:38.637146 23368 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260927 21:32:38.638685 23368 webserver.cc:460] Webserver started at http://127.0.97.60:44217/ using document root <none> and password file <none>
I20260927 21:32:38.639359 23368 fs_manager.cc:359] Metadata directory not provided
I20260927 21:32:38.639443 23368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:32:38.639696 23368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:32:38.642863 23368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-2/data/instance:
uuid: "9c7d167e4f1e4964928ad4fe18e18b41"
format_stamp: "Formatted at 2026-09-27 21:32:38 on dist-test-slave-bxm0"
server_key: "5b12573352ff08a660132c38aeb8f6d0"
server_key_iv: "c10439b0db030f30fe0ed7c752656ee1"
server_key_version: "encryptionkey@0"
I20260927 21:32:38.643589 23368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-2/wal/instance:
uuid: "9c7d167e4f1e4964928ad4fe18e18b41"
format_stamp: "Formatted at 2026-09-27 21:32:38 on dist-test-slave-bxm0"
server_key: "5b12573352ff08a660132c38aeb8f6d0"
server_key_iv: "c10439b0db030f30fe0ed7c752656ee1"
server_key_version: "encryptionkey@0"
I20260927 21:32:38.648780 23368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 21:32:38.652468 23383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:32:38.653977 23368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 21:32:38.654204 23368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-2/data,/tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-2/wal
uuid: "9c7d167e4f1e4964928ad4fe18e18b41"
format_stamp: "Formatted at 2026-09-27 21:32:38 on dist-test-slave-bxm0"
server_key: "5b12573352ff08a660132c38aeb8f6d0"
server_key_iv: "c10439b0db030f30fe0ed7c752656ee1"
server_key_version: "encryptionkey@0"
I20260927 21:32:38.654436 23368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:32:38.672407 23368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:32:38.677173 23368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:32:38.677464 23368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:32:38.696794 23368 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:36099
I20260927 21:32:38.696812 23434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:36099 every 8 connection(s)
I20260927 21:32:38.698325 23368 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-2/data/info.pb
I20260927 21:32:38.702170 23435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:32:38.705245 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu as pid 23368
I20260927 21:32:38.705440 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklB5kC7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790544390748963-388-0/minicluster-data/master-2/wal/instance
I20260927 21:32:38.706018 388 external_mini_cluster.cc:1428] Setting key 71387d1978d5228c4a3906128492dcfa
I20260927 21:32:38.710209 23435 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34953 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:32:38 dist-test-slave-bxm0 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 21:32:38 dist-test-slave-bxm0 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:32:38.729082 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.713804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36296 (local address 127.0.97.62:34953)
0927 21:32:38.714355 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:38.714362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:38.714884 (+ 522us) server_negotiation.cc:407] Connection header received
0927 21:32:38.715074 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:38.715079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:38.715147 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:38.715243 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:38.716486 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.717440 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.719268 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.719958 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.720151 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:38.723491 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:38.723495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:38.723498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:38.723531 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:38.726220 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.726929 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.726936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.726944 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.727068 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.727459 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.727468 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.727470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.727733 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:38.727902 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:38.728416 (+ 514us) server_negotiation.cc:299] Negotiation successful
0927 21:32:38.728643 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":265,"threads_started":1}
I20260927 21:32:38.729995 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.713854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34953 (local address 127.0.0.1:36296)
0927 21:32:38.714724 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.714767 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:32:38.714948 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:32:38.715336 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:32:38.715342 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:32:38.716064 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:32:38.716314 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:38.716324 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.717637 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.717643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:38.719119 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:38.719131 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.720213 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.720218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:38.720341 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:38.720984 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:32:38.721006 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:32:38.723287 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:32:38.726362 (+ 3075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:38.726375 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:38.726387 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:38.726798 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:38.727202 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:38.727207 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:38.727210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:38.727337 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:38.727876 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:32:38.727883 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:32:38.728087 (+ 204us) client_negotiation.cc:769] Sending connection context
0927 21:32:38.728303 (+ 216us) client_negotiation.cc:240] Negotiation successful
0927 21:32:38.728901 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":154,"thread_start_us":443,"threads_started":1}
I20260927 21:32:38.730013 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.708124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39892 (local address 127.0.97.60:36099)
0927 21:32:38.708925 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:38.708938 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:38.708980 (+ 42us) server_negotiation.cc:407] Connection header received
0927 21:32:38.710795 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:38.710832 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:38.711281 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:38.711678 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:38.713408 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.715479 (+ 2071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.717092 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.718025 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.718220 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:38.722733 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:38.722745 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:38.722758 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:38.722788 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:38.725786 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.726470 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.726482 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.726484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.726644 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.727019 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.727026 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.727029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.727562 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:38.727856 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:38.728268 (+ 412us) server_negotiation.cc:299] Negotiation successful
0927 21:32:38.728717 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":390,"threads_started":1}
I20260927 21:32:38.731709 23435 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44775 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:32:38 dist-test-slave-bxm0 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 27 21:32:38 dist-test-slave-bxm0 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 21:32:38.746152 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.732578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44775 (local address 127.0.0.1:40294)
0927 21:32:38.732757 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.732776 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:32:38.732886 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:32:38.733379 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:32:38.733382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:32:38.733794 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:32:38.734009 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:38.734021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.735133 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.735136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:38.736347 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:38.736362 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.737184 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.737190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:38.737417 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:38.738088 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:32:38.738105 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:32:38.740252 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:32:38.743673 (+ 3421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:38.743679 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:38.743683 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:38.744061 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:38.744459 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:38.744465 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:38.744468 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:38.744616 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:38.745137 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:32:38.745142 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:32:38.745302 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 21:32:38.745446 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 21:32:38.745774 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 21:32:38.746178 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.732710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40294 (local address 127.0.97.61:44775)
0927 21:32:38.732921 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:38.732929 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:38.732951 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:32:38.733016 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:38.733021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:38.733113 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:38.733215 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:38.734237 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.734946 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.736530 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.737037 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.737132 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:38.740465 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:38.740470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:38.740473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:38.740507 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:38.743520 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.744194 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.744200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.744204 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.744343 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.744730 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.744736 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.744739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.745012 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:38.745176 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:38.745533 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 21:32:38.745791 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 21:32:38.747258 23435 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:32:38.747870 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.733199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36304 (local address 127.0.97.62:34953)
0927 21:32:38.733535 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:38.733540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:38.733552 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:32:38.733590 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:38.733594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:38.733735 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:38.733809 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:38.735063 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.735817 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.737430 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.737926 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.738062 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:38.742406 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:38.742412 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:38.742415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:38.742449 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:38.745069 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.745834 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.745840 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.745849 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.745968 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.746410 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.746417 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.746423 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.746767 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:38.747059 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:38.747360 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 21:32:38.747603 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245}
Sep 27 21:32:38 dist-test-slave-bxm0 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 21:32:38 dist-test-slave-bxm0 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 21:32:38.764206 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.748234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36099 (local address 127.0.0.1:39906)
0927 21:32:38.748802 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.748817 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:32:38.748951 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:32:38.749290 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:32:38.749295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:32:38.749801 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:32:38.749999 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:38.750012 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.751379 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.751383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:38.752558 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:38.752572 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.753482 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.753486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:38.753746 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:38.754335 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:32:38.754352 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:32:38.756618 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:32:38.760894 (+ 4276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:38.760898 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:38.760902 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:38.761380 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:38.761818 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:38.761822 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:38.761825 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:38.761983 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:38.763024 (+ 1041us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:32:38.763030 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:32:38.763189 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 21:32:38.763376 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 21:32:38.763707 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479}
I20260927 21:32:38.764392 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.748154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39906 (local address 127.0.97.60:36099)
0927 21:32:38.748357 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:38.748363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:38.748891 (+ 528us) server_negotiation.cc:407] Connection header received
0927 21:32:38.749065 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:38.749072 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:38.749148 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:38.749240 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:38.750151 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.751246 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.752735 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.753326 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.753473 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:38.756806 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:38.756814 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:38.756819 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:38.756856 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:38.760732 (+ 3876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.761514 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.761520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.761525 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.761698 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.762110 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.762116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.762121 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.762533 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:38.762697 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:38.763504 (+ 807us) server_negotiation.cc:299] Negotiation successful
0927 21:32:38.763954 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":125,"server-negotiator.queue_time_us":86}
I20260927 21:32:38.768957 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.749974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40304 (local address 127.0.97.61:44775)
0927 21:32:38.750147 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:38.750153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:38.750181 (+ 28us) server_negotiation.cc:407] Connection header received
0927 21:32:38.750282 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:38.750286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:38.750337 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:38.750407 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:38.751843 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.752852 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.754319 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.754740 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.754879 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:38.760977 (+ 6098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:38.760983 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:38.760989 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:38.761031 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:38.766199 (+ 5168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.766928 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.766934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.766937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.767110 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.767506 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.767511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.767515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.767858 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:38.768129 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:38.768359 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 21:32:38.768604 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 21:32:38.771253 23435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c7d167e4f1e4964928ad4fe18e18b41: Bootstrap starting.
I20260927 21:32:38.774351 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:32:38.775431 23435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c7d167e4f1e4964928ad4fe18e18b41: Neither blocks nor log segments found. Creating new log.
I20260927 21:32:38.776789 23435 log.cc:824] T 00000000000000000000000000000000 P 9c7d167e4f1e4964928ad4fe18e18b41: Log is configured to *not* fsync() on all Append() calls
I20260927 21:32:38.780189 23435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c7d167e4f1e4964928ad4fe18e18b41: No bootstrap required, opened a new log
I20260927 21:32:38.777647 23262 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "191dc9d656e3456cba924abf9b41c441"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36304
I20260927 21:32:38.791348 23435 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c7d167e4f1e4964928ad4fe18e18b41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191dc9d656e3456cba924abf9b41c441" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34953 } } peers { permanent_uuid: "e69011d81fe843248c42fc696460fd64" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44775 } } peers { permanent_uuid: "9c7d167e4f1e4964928ad4fe18e18b41" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } }
I20260927 21:32:38.791805 23435 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c7d167e4f1e4964928ad4fe18e18b41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c7d167e4f1e4964928ad4fe18e18b41, State: Initialized, Role: FOLLOWER
I20260927 21:32:38.792325 23435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c7d167e4f1e4964928ad4fe18e18b41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191dc9d656e3456cba924abf9b41c441" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34953 } } peers { permanent_uuid: "e69011d81fe843248c42fc696460fd64" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44775 } } peers { permanent_uuid: "9c7d167e4f1e4964928ad4fe18e18b41" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } }
I20260927 21:32:38.793303 23442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c7d167e4f1e4964928ad4fe18e18b41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191dc9d656e3456cba924abf9b41c441" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34953 } } peers { permanent_uuid: "e69011d81fe843248c42fc696460fd64" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44775 } } peers { permanent_uuid: "9c7d167e4f1e4964928ad4fe18e18b41" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } } }
I20260927 21:32:38.793689 23442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c7d167e4f1e4964928ad4fe18e18b41 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:32:38.794574 23435 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c7d167e4f1e4964928ad4fe18e18b41 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 21:32:38.804358 23453 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c7d167e4f1e4964928ad4fe18e18b41: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:32:38.804503 23453 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:32:38.801976 23368 master_runner.cc:186] Error getting master registration for 127.0.97.62:34953: OK, instance_id { permanent_uuid: "191dc9d656e3456cba924abf9b41c441" instance_seqno: 1790544757670192 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:32:38.809587 23368 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:32:38.813091 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } attempt: 4
Sep 27 21:32:38 dist-test-slave-bxm0 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 21:32:38.828454 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.813860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36099 (local address 127.0.0.1:39918)
0927 21:32:38.814071 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.814094 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 21:32:38.814280 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:32:38.814633 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:32:38.814638 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:32:38.815049 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:32:38.815246 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:38.815256 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.816329 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.816337 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:38.817251 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:38.817263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.818122 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.818128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:38.818329 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:38.818975 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:32:38.818993 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:32:38.821427 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:32:38.825475 (+ 4048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:38.825481 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:38.825485 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:38.825950 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:38.826387 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:38.826393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:38.826398 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:38.826594 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:38.827328 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:32:38.827334 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:32:38.827521 (+ 187us) client_negotiation.cc:769] Sending connection context
0927 21:32:38.827691 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 21:32:38.828041 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 21:32:38.828469 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.813947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39918 (local address 127.0.97.60:36099)
0927 21:32:38.814140 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:38.814146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:38.814167 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:32:38.814422 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:38.814427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:38.814500 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:38.814605 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:38.815390 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.816180 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.817403 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.817967 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.818094 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:38.821650 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:38.821660 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:38.821673 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:38.821710 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:38.825327 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.826078 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.826084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.826089 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.826239 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.826718 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.826726 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.826731 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.827172 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:38.827340 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:38.827814 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 21:32:38.828048 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 21:32:38.833029 23287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441: Bootstrap starting.
I20260927 21:32:38.836885 23287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441: Neither blocks nor log segments found. Creating new log.
I20260927 21:32:38.838059 23287 log.cc:824] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441: Log is configured to *not* fsync() on all Append() calls
I20260927 21:32:38.841316 23287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441: No bootstrap required, opened a new log
I20260927 21:32:38.846800 23287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191dc9d656e3456cba924abf9b41c441" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34953 } } peers { permanent_uuid: "e69011d81fe843248c42fc696460fd64" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44775 } } peers { permanent_uuid: "9c7d167e4f1e4964928ad4fe18e18b41" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } }
I20260927 21:32:38.847210 23287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 191dc9d656e3456cba924abf9b41c441, State: Initialized, Role: FOLLOWER
I20260927 21:32:38.847698 23287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191dc9d656e3456cba924abf9b41c441" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34953 } } peers { permanent_uuid: "e69011d81fe843248c42fc696460fd64" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44775 } } peers { permanent_uuid: "9c7d167e4f1e4964928ad4fe18e18b41" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } }
I20260927 21:32:38.848650 23456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191dc9d656e3456cba924abf9b41c441" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34953 } } peers { permanent_uuid: "e69011d81fe843248c42fc696460fd64" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44775 } } peers { permanent_uuid: "9c7d167e4f1e4964928ad4fe18e18b41" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } } }
I20260927 21:32:38.849058 23456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:32:38.849926 23287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:32:38.850106 23262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 21:32:38.850342 23262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:32:38.854228 23262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191dc9d656e3456cba924abf9b41c441" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34953 } } peers { permanent_uuid: "e69011d81fe843248c42fc696460fd64" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44775 } } peers { permanent_uuid: "9c7d167e4f1e4964928ad4fe18e18b41" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } }
I20260927 21:32:38.858970 23410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "191dc9d656e3456cba924abf9b41c441" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9c7d167e4f1e4964928ad4fe18e18b41"
I20260927 21:32:38.859481 23410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c7d167e4f1e4964928ad4fe18e18b41 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:32:38.863860 23410 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9c7d167e4f1e4964928ad4fe18e18b41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191dc9d656e3456cba924abf9b41c441 in term 1.
I20260927 21:32:38.864529 23239 leader_election.cc:302] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 191dc9d656e3456cba924abf9b41c441, 9c7d167e4f1e4964928ad4fe18e18b41; no voters:
I20260927 21:32:38.865114 23456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:32:38.858083 23262 leader_election.cc:288] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441 [CANDIDATE]: Term 1 election: Requested vote from peers e69011d81fe843248c42fc696460fd64 (127.0.97.61:44775), 9c7d167e4f1e4964928ad4fe18e18b41 (127.0.97.60:36099)
I20260927 21:32:38.875567 23456 raft_consensus.cc:686] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441 [term 1 LEADER]: Becoming Leader. State: Replica: 191dc9d656e3456cba924abf9b41c441, State: Running, Role: LEADER
I20260927 21:32:38.876291 23456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191dc9d656e3456cba924abf9b41c441" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34953 } } peers { permanent_uuid: "e69011d81fe843248c42fc696460fd64" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44775 } } peers { permanent_uuid: "9c7d167e4f1e4964928ad4fe18e18b41" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } }
W20260927 21:32:38.884768 23467 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:32:38.884939 23467 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:32:38.887027 23469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 191dc9d656e3456cba924abf9b41c441. Latest consensus state: current_term: 1 leader_uuid: "191dc9d656e3456cba924abf9b41c441" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191dc9d656e3456cba924abf9b41c441" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34953 } } peers { permanent_uuid: "e69011d81fe843248c42fc696460fd64" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44775 } } peers { permanent_uuid: "9c7d167e4f1e4964928ad4fe18e18b41" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } } }
I20260927 21:32:38.887697 23469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:38.892037 23473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:32:38.903609 23361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } attempt: 5
I20260927 21:32:38.863422 23220 master_runner.cc:186] Error getting master registration for 127.0.97.61:44775: OK, instance_id { permanent_uuid: "e69011d81fe843248c42fc696460fd64" instance_seqno: 1790544758170243 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:32:38.905644 23473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:32:38.907389 23220 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 21:32:38.909526 23241 tablet.cc:2369] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 21:32:38 dist-test-slave-bxm0 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790544758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260927 21:32:38.929814 23389 tablet.cc:2369] T 00000000000000000000000000000000 P 9c7d167e4f1e4964928ad4fe18e18b41: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:32:38.939345 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.893648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40314 (local address 127.0.97.61:44775)
0927 21:32:38.893852 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:38.893860 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:38.906358 (+ 12498us) server_negotiation.cc:407] Connection header received
0927 21:32:38.906673 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:38.906682 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:38.906801 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:38.906892 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:38.912476 (+ 5584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.914001 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.920447 (+ 6446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.921210 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.921337 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:38.931919 (+ 10582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:38.931929 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:38.931935 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:38.931975 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:38.935702 (+ 3727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.936625 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.936631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.936634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.936796 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.937332 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.937339 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.937345 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.937750 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:38.938010 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:38.938646 (+ 636us) server_negotiation.cc:299] Negotiation successful
0927 21:32:38.938884 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:32:38.943362 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.904242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36099 (local address 127.0.0.1:39928)
0927 21:32:38.904634 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 21:32:38.904655 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:32:38.904773 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:32:38.918190 (+ 13417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:32:38.918195 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:32:38.918740 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:32:38.918968 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:38.918981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.920719 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.920727 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:38.922279 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:32:38.922298 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.923411 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.923417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:32:38.923663 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:38.924314 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:32:38.924339 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:32:38.927249 (+ 2910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:32:38.931953 (+ 4704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:38.931958 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:38.931961 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:38.932821 (+ 860us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:38.941395 (+ 8574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:32:38.941400 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:32:38.941403 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:32:38.941582 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:32:38.942302 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:32:38.942309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:32:38.942467 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 21:32:38.942614 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 21:32:38.942943 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252}
I20260927 21:32:38.944689 23479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.905011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39928 (local address 127.0.97.60:36099)
0927 21:32:38.917725 (+ 12714us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:38.917732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:38.917755 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:32:38.917811 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:38.917816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:38.917893 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:38.919134 (+ 1241us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:38.919411 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.920570 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.922491 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.923215 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.927407 (+ 4192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:38.927671 (+ 264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:38.927678 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:38.927682 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:38.927715 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:38.931820 (+ 4105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.940142 (+ 8322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.940149 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.940152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.940327 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.941766 (+ 1439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.941771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.941774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.942145 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:38.943757 (+ 1612us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:38.943924 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 21:32:38.944200 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12547,"thread_start_us":12967,"threads_started":1}
I20260927 21:32:38.947531 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.893303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36308 (local address 127.0.97.62:34953)
0927 21:32:38.893509 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:38.893523 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:38.897716 (+ 4193us) server_negotiation.cc:407] Connection header received
0927 21:32:38.897955 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:38.897964 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:38.898078 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:38.898217 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:38.916681 (+ 18464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.917955 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.927230 (+ 9275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.927990 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.928133 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:38.937161 (+ 9028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:38.937168 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:38.937173 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:38.937221 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:38.941024 (+ 3803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.944966 (+ 3942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.944972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.944975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.945150 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.945648 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.945653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.945657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.946064 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:38.946357 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:38.946893 (+ 536us) server_negotiation.cc:299] Negotiation successful
0927 21:32:38.947148 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 21:32:38.950639 23361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e69011d81fe843248c42fc696460fd64: Bootstrap starting.
I20260927 21:32:38.955081 23361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e69011d81fe843248c42fc696460fd64: Neither blocks nor log segments found. Creating new log.
I20260927 21:32:38.956395 23361 log.cc:824] T 00000000000000000000000000000000 P e69011d81fe843248c42fc696460fd64: Log is configured to *not* fsync() on all Append() calls
I20260927 21:32:38.961227 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:38.896827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39920 (local address 127.0.97.60:36099)
0927 21:32:38.897026 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:38.897035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:38.924923 (+ 27888us) server_negotiation.cc:407] Connection header received
0927 21:32:38.925182 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:38.925187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:38.925268 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:38.925337 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:38.928028 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.929272 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.935249 (+ 5977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:38.935979 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:38.936100 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:38.949797 (+ 13697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:32:38.949804 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:32:38.949809 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:32:38.949854 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:32:38.953753 (+ 3899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.957775 (+ 4022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.957783 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.957786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.958009 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:32:38.958642 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:32:38.958652 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:32:38.958658 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:32:38.959162 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:32:38.959497 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:38.960266 (+ 769us) server_negotiation.cc:299] Negotiation successful
0927 21:32:38.960634 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 21:32:38.965359 23410 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c7d167e4f1e4964928ad4fe18e18b41 [term 1 FOLLOWER]: Refusing update from remote peer 191dc9d656e3456cba924abf9b41c441: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:32:38.966490 23456 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c7d167e4f1e4964928ad4fe18e18b41" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:32:38.968834 23361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e69011d81fe843248c42fc696460fd64: No bootstrap required, opened a new log
I20260927 21:32:38.981235 23361 raft_consensus.cc:348] T 00000000000000000000000000000000 P e69011d81fe843248c42fc696460fd64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191dc9d656e3456cba924abf9b41c441" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34953 } } peers { permanent_uuid: "e69011d81fe843248c42fc696460fd64" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44775 } } peers { permanent_uuid: "9c7d167e4f1e4964928ad4fe18e18b41" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } }
I20260927 21:32:38.981724 23361 raft_consensus.cc:729] T 00000000000000000000000000000000 P e69011d81fe843248c42fc696460fd64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e69011d81fe843248c42fc696460fd64, State: Initialized, Role: FOLLOWER
I20260927 21:32:38.982282 23361 consensus_queue.cc:260] T 00000000000000000000000000000000 P e69011d81fe843248c42fc696460fd64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191dc9d656e3456cba924abf9b41c441" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34953 } } peers { permanent_uuid: "e69011d81fe843248c42fc696460fd64" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44775 } } peers { permanent_uuid: "9c7d167e4f1e4964928ad4fe18e18b41" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } }
I20260927 21:32:38.994028 23442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c7d167e4f1e4964928ad4fe18e18b41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 191dc9d656e3456cba924abf9b41c441. Latest consensus state: current_term: 1 leader_uuid: "191dc9d656e3456cba924abf9b41c441" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191dc9d656e3456cba924abf9b41c441" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34953 } } peers { permanent_uuid: "e69011d81fe843248c42fc696460fd64" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44775 } } peers { permanent_uuid: "9c7d167e4f1e4964928ad4fe18e18b41" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } } }
I20260927 21:32:38.994433 23442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c7d167e4f1e4964928ad4fe18e18b41 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:32:38.997581 23480 mvcc.cc:205] Tried to move back new op lower bound from 7334071332685099008 to 7334071332363808768. Current Snapshot: MvccSnapshot[applied={T|T < 7334071332685099008}]
I20260927 21:32:39.002110 23456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "191dc9d656e3456cba924abf9b41c441" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191dc9d656e3456cba924abf9b41c441" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34953 } } peers { permanent_uuid: "e69011d81fe843248c42fc696460fd64" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44775 } } peers { permanent_uuid: "9c7d167e4f1e4964928ad4fe18e18b41" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } } }
I20260927 21:32:39.002427 23456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:39.006358 23482 mvcc.cc:205] Tried to move back new op lower bound from 7334071332685099008 to 7334071332363808768. Current Snapshot: MvccSnapshot[applied={T|T < 7334071332685099008}]
I20260927 21:32:39.011349 23473 catalog_manager.cc:1347] Generated new cluster ID: 376a101aee924dc4bc55cf2b93416851
I20260927 21:32:39.011464 23473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:32:39.012032 23442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c7d167e4f1e4964928ad4fe18e18b41 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "191dc9d656e3456cba924abf9b41c441" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191dc9d656e3456cba924abf9b41c441" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34953 } } peers { permanent_uuid: "e69011d81fe843248c42fc696460fd64" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44775 } } peers { permanent_uuid: "9c7d167e4f1e4964928ad4fe18e18b41" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } } }
I20260927 21:32:39.012408 23442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c7d167e4f1e4964928ad4fe18e18b41 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:32:39.018383 23484 sys_catalog.cc:455] T 00000000000000000000000000000000 P e69011d81fe843248c42fc696460fd64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191dc9d656e3456cba924abf9b41c441" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34953 } } peers { permanent_uuid: "e69011d81fe843248c42fc696460fd64" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44775 } } peers { permanent_uuid: "9c7d167e4f1e4964928ad4fe18e18b41" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } } }
I20260927 21:32:39.018980 23484 sys_catalog.cc:458] T 00000000000000000000000000000000 P e69011d81fe843248c42fc696460fd64 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:32:39.019758 23361 sys_catalog.cc:564] T 00000000000000000000000000000000 P e69011d81fe843248c42fc696460fd64 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:32:39.020123 23335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e69011d81fe843248c42fc696460fd64 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:32:39.023856 23335 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e69011d81fe843248c42fc696460fd64 [term 1 FOLLOWER]: Refusing update from remote peer 191dc9d656e3456cba924abf9b41c441: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:32:39.024453 23336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "191dc9d656e3456cba924abf9b41c441" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e69011d81fe843248c42fc696460fd64"
I20260927 21:32:39.024956 23469 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e69011d81fe843248c42fc696460fd64" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:32:39.028180 23336 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e69011d81fe843248c42fc696460fd64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 191dc9d656e3456cba924abf9b41c441 in term 1.
I20260927 21:32:39.041584 23473 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:32:39.043445 23473 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:32:39.077817 23484 sys_catalog.cc:455] T 00000000000000000000000000000000 P e69011d81fe843248c42fc696460fd64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 191dc9d656e3456cba924abf9b41c441. Latest consensus state: current_term: 1 leader_uuid: "191dc9d656e3456cba924abf9b41c441" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191dc9d656e3456cba924abf9b41c441" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34953 } } peers { permanent_uuid: "e69011d81fe843248c42fc696460fd64" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44775 } } peers { permanent_uuid: "9c7d167e4f1e4964928ad4fe18e18b41" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } } }
I20260927 21:32:39.078177 23484 sys_catalog.cc:458] T 00000000000000000000000000000000 P e69011d81fe843248c42fc696460fd64 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:32:39.078311 23473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441: Generated new TSK 0
I20260927 21:32:39.078521 23494 sys_catalog.cc:455] T 00000000000000000000000000000000 P e69011d81fe843248c42fc696460fd64 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "191dc9d656e3456cba924abf9b41c441" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191dc9d656e3456cba924abf9b41c441" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34953 } } peers { permanent_uuid: "e69011d81fe843248c42fc696460fd64" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44775 } } peers { permanent_uuid: "9c7d167e4f1e4964928ad4fe18e18b41" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } } }
I20260927 21:32:39.078902 23494 sys_catalog.cc:458] T 00000000000000000000000000000000 P e69011d81fe843248c42fc696460fd64 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:32:39.079422 23473 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:32:39.080511 23456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "191dc9d656e3456cba924abf9b41c441" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "191dc9d656e3456cba924abf9b41c441" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34953 } } peers { permanent_uuid: "e69011d81fe843248c42fc696460fd64" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44775 } } peers { permanent_uuid: "9c7d167e4f1e4964928ad4fe18e18b41" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36099 } } }
I20260927 21:32:39.080824 23456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 191dc9d656e3456cba924abf9b41c441 [sys.catalog]: This master's current role is: LEADER
I20260927 21:32:39.111076 23501 catalog_manager.cc:1259] Loaded cluster ID: 376a101aee924dc4bc55cf2b93416851
I20260927 21:32:39.111199 23501 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e69011d81fe843248c42fc696460fd64: loading cluster ID for follower catalog manager: success
I20260927 21:32:39.114879 23501 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e69011d81fe843248c42fc696460fd64: acquiring CA information for follower catalog manager: success
I20260927 21:32:39.117589 23501 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e69011d81fe843248c42fc696460fd64: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 21:32:39.235565 23438 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:32:39.227058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39936 (local address 127.0.97.60:36099)
0927 21:32:39.227278 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:39.227288 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:39.233007 (+ 5719us) server_negotiation.cc:407] Connection header received
0927 21:32:39.233832 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:39.233840 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:39.233940 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:39.234020 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:39.235217 (+ 1197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39936: BlockingRecv error: recv got EOF from 127.0.0.1:39936 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 21:32:39.238765 23439 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:32:39.220037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36318 (local address 127.0.97.62:34953)
0927 21:32:39.220256 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:39.220266 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:39.232561 (+ 12295us) server_negotiation.cc:407] Connection header received
0927 21:32:39.236433 (+ 3872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:39.236439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:39.236526 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:39.236599 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:39.238492 (+ 1893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36318: BlockingRecv error: recv got EOF from 127.0.0.1:36318 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 21:32:39.240202 23365 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:32:39.214847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40326 (local address 127.0.97.61:44775)
0927 21:32:39.215045 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:39.215052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:39.231703 (+ 16651us) server_negotiation.cc:407] Connection header received
0927 21:32:39.236023 (+ 4320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:39.236030 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:39.236124 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:39.236200 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:39.239947 (+ 3747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40326: BlockingRecv error: recv got EOF from 127.0.0.1:40326 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 21:32:39.293426 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:39.274077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40330 (local address 127.0.97.61:44775)
0927 21:32:39.274270 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:39.274277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:39.285096 (+ 10819us) server_negotiation.cc:407] Connection header received
0927 21:32:39.285712 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:39.285719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:39.285984 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:39.286143 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:32:39.286888 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:39.288087 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:39.290250 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:39.290974 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:39.291074 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:39.291860 (+ 786us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:32:39.292578 (+ 718us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:32:39.292712 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:39.292966 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 21:32:39.293067 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 21:32:39.339414 23438 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:32:39.329978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39946 (local address 127.0.97.60:36099)
0927 21:32:39.330212 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:39.330220 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:39.337279 (+ 7059us) server_negotiation.cc:407] Connection header received
0927 21:32:39.337866 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:39.337873 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:39.338003 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:39.338086 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:39.339176 (+ 1090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39946: BlockingRecv error: recv got EOF from 127.0.0.1:39946 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 21:32:39.391433 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:39.373446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40342 (local address 127.0.97.61:44775)
0927 21:32:39.373676 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:39.373682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:39.384344 (+ 10662us) server_negotiation.cc:407] Connection header received
0927 21:32:39.384885 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:39.384892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:39.384981 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:39.385054 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:32:39.385763 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:39.386963 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:39.389067 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:39.389809 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:39.389906 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:39.390523 (+ 617us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:32:39.390707 (+ 184us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:32:39.390769 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:39.391002 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 21:32:39.391093 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 21:32:39.434074 23438 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:32:39.424913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39956 (local address 127.0.97.60:36099)
0927 21:32:39.425105 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:39.425112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:39.432037 (+ 6925us) server_negotiation.cc:407] Connection header received
0927 21:32:39.432535 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:39.432541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:39.432668 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:39.432747 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:39.433818 (+ 1071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39956: BlockingRecv error: recv got EOF from 127.0.0.1:39956 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 21:32:39.467664 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:39.459710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40344 (local address 127.0.97.61:44775)
0927 21:32:39.459912 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:39.459919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:39.460853 (+ 934us) server_negotiation.cc:407] Connection header received
0927 21:32:39.461080 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:39.461087 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:39.461184 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:39.461289 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:32:39.461891 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:39.463128 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:39.465230 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:39.465987 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:39.466128 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:39.466705 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:32:39.466882 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:32:39.466971 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:39.467204 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 21:32:39.467312 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 21:32:39.477890 23438 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:32:39.475388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39960 (local address 127.0.97.60:36099)
0927 21:32:39.475614 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:39.475621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:39.476234 (+ 613us) server_negotiation.cc:407] Connection header received
0927 21:32:39.476453 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:39.476459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:39.476555 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:39.476675 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:39.477601 (+ 926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39960: BlockingRecv error: recv got EOF from 127.0.0.1:39960 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 21:32:39.492642 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:39.485485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40346 (local address 127.0.97.61:44775)
0927 21:32:39.485715 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:39.485721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:39.486339 (+ 618us) server_negotiation.cc:407] Connection header received
0927 21:32:39.486552 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:39.486557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:39.486638 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:39.486741 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:32:39.487284 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:39.488407 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:39.490465 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:39.491092 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:39.491222 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:39.491748 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:32:39.491902 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:32:39.491972 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:39.492246 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 21:32:39.492347 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 21:32:39.506740 23438 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:32:39.503926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39972 (local address 127.0.97.60:36099)
0927 21:32:39.504172 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:39.504177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:39.504983 (+ 806us) server_negotiation.cc:407] Connection header received
0927 21:32:39.505301 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:39.505307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:39.505425 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:39.505539 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:39.506501 (+ 962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39972: BlockingRecv error: recv got EOF from 127.0.0.1:39972 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
I20260927 21:32:39.525604 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:39.518313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40354 (local address 127.0.97.61:44775)
0927 21:32:39.518511 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:39.518519 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:39.519084 (+ 565us) server_negotiation.cc:407] Connection header received
0927 21:32:39.519286 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:39.519293 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:39.519393 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:39.519503 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:32:39.520078 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:39.521314 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:39.523341 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:39.524066 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:39.524216 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:39.524725 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:32:39.524944 (+ 219us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:32:39.525039 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:39.525202 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 21:32:39.525288 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 21:32:39.539960 23438 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:32:39.535802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39976 (local address 127.0.97.60:36099)
0927 21:32:39.535993 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:39.535999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:39.538360 (+ 2361us) server_negotiation.cc:407] Connection header received
0927 21:32:39.538569 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:39.538575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:39.538683 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:39.538794 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:39.539719 (+ 925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39976: BlockingRecv error: recv got EOF from 127.0.0.1:39976 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 21:32:39.566609 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:39.559360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40362 (local address 127.0.97.61:44775)
0927 21:32:39.559545 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:39.559551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:39.560068 (+ 517us) server_negotiation.cc:407] Connection header received
0927 21:32:39.560274 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:39.560280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:39.560390 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:39.560511 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:32:39.560959 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:39.562170 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:39.564733 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:39.565195 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:39.565300 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:39.565904 (+ 604us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:32:39.566018 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:32:39.566088 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:39.566261 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 21:32:39.566328 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 21:32:39.574800 23438 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:32:39.572842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39982 (local address 127.0.97.60:36099)
0927 21:32:39.573011 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:39.573015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:39.573461 (+ 446us) server_negotiation.cc:407] Connection header received
0927 21:32:39.573587 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:39.573595 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:39.573728 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:39.573839 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:39.574559 (+ 720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39982: BlockingRecv error: recv got EOF from 127.0.0.1:39982 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:32:39.616437 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:39.610203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40368 (local address 127.0.97.61:44775)
0927 21:32:39.610439 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:39.610446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:39.610814 (+ 368us) server_negotiation.cc:407] Connection header received
0927 21:32:39.611018 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:39.611024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:39.611123 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:39.611240 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:32:39.611639 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:39.612845 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:39.614312 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:39.615036 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:39.615176 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:39.615576 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:32:39.615758 (+ 182us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:32:39.615858 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:39.615988 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 21:32:39.616074 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260927 21:32:39.627986 23438 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:32:39.623411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39984 (local address 127.0.97.60:36099)
0927 21:32:39.623817 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:39.623827 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:39.626070 (+ 2243us) server_negotiation.cc:407] Connection header received
0927 21:32:39.626240 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:39.626246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:39.626345 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:39.626458 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:39.627754 (+ 1296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39984: BlockingRecv error: recv got EOF from 127.0.0.1:39984 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 21:32:39.701504 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:39.695681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40372 (local address 127.0.97.61:44775)
0927 21:32:39.695939 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:39.695945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:39.696220 (+ 275us) server_negotiation.cc:407] Connection header received
0927 21:32:39.696506 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:39.696510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:39.696588 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:39.696663 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:32:39.697183 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:39.697980 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:39.699866 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:39.700271 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:39.700367 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:39.700853 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:32:39.700948 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:32:39.701010 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:39.701180 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 21:32:39.701243 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260927 21:32:39.709378 23438 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:32:39.707432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39994 (local address 127.0.97.60:36099)
0927 21:32:39.707668 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:39.707673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:39.707962 (+ 289us) server_negotiation.cc:407] Connection header received
0927 21:32:39.708120 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:39.708123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:39.708196 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:39.708277 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:32:39.709121 (+ 844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39994: BlockingRecv error: recv got EOF from 127.0.0.1:39994 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 21:32:39.806536 23453 catalog_manager.cc:1259] Loaded cluster ID: 376a101aee924dc4bc55cf2b93416851
I20260927 21:32:39.806686 23453 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9c7d167e4f1e4964928ad4fe18e18b41: loading cluster ID for follower catalog manager: success
I20260927 21:32:39.809111 23453 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9c7d167e4f1e4964928ad4fe18e18b41: acquiring CA information for follower catalog manager: success
I20260927 21:32:39.810947 23453 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9c7d167e4f1e4964928ad4fe18e18b41: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:32:39.845546 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:39.840792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40382 (local address 127.0.97.61:44775)
0927 21:32:39.841001 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:39.841009 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:39.841433 (+ 424us) server_negotiation.cc:407] Connection header received
0927 21:32:39.841560 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:39.841564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:39.841668 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:39.841769 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:32:39.842151 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:39.842853 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:39.844104 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:39.844487 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:39.844582 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:39.844943 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:32:39.845052 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:32:39.845117 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:39.845271 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 21:32:39.845335 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:32:39.857854 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:39.852159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40004 (local address 127.0.97.60:36099)
0927 21:32:39.852352 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:39.852357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:39.852814 (+ 457us) server_negotiation.cc:407] Connection header received
0927 21:32:39.852955 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:39.852959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:39.853151 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:39.853287 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:32:39.853662 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:39.854514 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:39.855773 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:39.856201 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:39.856290 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:39.856729 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:32:39.857268 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:32:39.857382 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:39.857451 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 21:32:39.857515 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 21:32:39.879029 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:39.865800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40384 (local address 127.0.97.61:44775)
0927 21:32:39.866042 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:39.866051 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:39.872784 (+ 6733us) server_negotiation.cc:407] Connection header received
0927 21:32:39.873100 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:39.873109 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:39.873230 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:39.873334 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:32:39.874291 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:39.875435 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:39.877045 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:39.877489 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:39.877671 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:39.878257 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:32:39.878363 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:32:39.878460 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:39.878631 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 21:32:39.878726 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 21:32:39.880978 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:39.865407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36324 (local address 127.0.97.62:34953)
0927 21:32:39.873447 (+ 8040us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:39.873455 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:39.873477 (+ 22us) server_negotiation.cc:407] Connection header received
0927 21:32:39.873653 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:39.873660 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:39.873902 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:39.874120 (+ 218us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:32:39.874828 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:39.875814 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:39.878276 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:39.879002 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:39.879143 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:39.879875 (+ 732us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:32:39.880284 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:32:39.880395 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:39.880577 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 21:32:39.880667 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7902,"thread_start_us":7790,"threads_started":1}
I20260927 21:32:39.881464 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:32:39.865406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40006 (local address 127.0.97.60:36099)
0927 21:32:39.865709 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0927 21:32:39.865719 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:32:39.874391 (+ 8672us) server_negotiation.cc:407] Connection header received
0927 21:32:39.874730 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:32:39.874737 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:32:39.874834 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:32:39.874938 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:32:39.875718 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:39.876825 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:39.879123 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:32:39.879723 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:32:39.879849 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:32:39.880370 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:32:39.880528 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:32:39.880604 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 21:32:39.881016 (+ 412us) server_negotiation.cc:299] Negotiation successful
0927 21:32:39.881104 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 21:32:39.892380 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 23220
I20260927 21:32:39.893383 23282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:32:40.005003 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 23220
W20260927 21:32:40.025687 23312 connection.cc:535] server connection from 127.0.0.1:40280 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 21:32:40.026037 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 23294
I20260927 21:32:40.027166 23356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:32:40.134685 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 23294
I20260927 21:32:40.155059 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu pid 23368
I20260927 21:32:40.156210 23430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 21:32:40.289655 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklB5kC7/build/asan/bin/kudu with pid 23368
2026-09-27T21:32:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4418 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8812 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (369496 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST